沧海一粟

天下事有难易乎?为之,则难者亦易矣;不为,则易者亦难矣。

0%

Elasticsearch Logstash Kibana 笔记

Elastic Stack 是适用于数据采集、充实、存储、分析和可视化的一组开源工具。人们通常将 Elastic Stack 称为 ELK Stack(代指 Elasticsearch、Logstash 和 Kibana)

Elasticsearch 是一个分布式的开源搜索和分析引擎,适用于所有类型的数据,包括文本、数字、地理空间、结构化和非结构化数据。 类似于MySQL,用来存储和分析数据。
Logstash可用来对数据进行聚合和处理,并将数据发送到 Elasticsearch。Logstash 是一个开源的服务器端数据处理管道,允许您在将数据索引到 Elasticsearch 之前同时从多个来源采集数据,并对数据进行充实和转换。 类似于程序,抽取解析数据并储存到ES
Kibana 是一款适用于 Elasticsearch 的数据可视化和管理工具。类似于使用MySQL时用的Navicat

简单的说,LogStash用来收取解析日志并发消息写入ES,Elasticsearch用来存储和分析查询,Kibana用来查看

es下载地址 https://www.elastic.co/cn/downloads/elasticsearch
logstash下载地址 https://www.elastic.co/cn/downloads/logstash
kibana下载地址 https://www.elastic.co/cn/downloads/kibana
filebeats

elasticsearch-7.x 要求JDK版本大于等于11

1
2
3
4
5
ZBMAC-C02PGMT0F:~ weikeqin1$ java -version
java version "14" 2020-03-17
Java(TM) SE Runtime Environment (build 14+36-1461)
Java HotSpot(TM) 64-Bit Server VM (build 14+36-1461, mixed mode, sharing)
ZBMAC-C02PGMT0F:~ weikeqin1$

elasticsearch安装配置启动

下载

1
2
3
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.7.0-linux-x86_64.tar.gz
tar -xzf elasticsearch-7.7.0-linux-x86_64.tar.gz
cd elasticsearch-7.7.0/

配置

修改 elasticsearch-7.7.0/config/elasticsearch.yml

1
2
action.destructive_requires_name: true
xpack.ml.enabled: true

启动

1
2
cd elasticsearch-7.7.0/
./bin/elasticsearch

后台启动可以用 ./bin/elasticsearch -d

验证是否启动成功

在浏览器里输入 http://localhost:9200/?pretty 或者用 curl 'http://localhost:9200/?pretty'

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
"name": "192.168.0.110",
"cluster_name": "elasticsearch",
"cluster_uuid": "aHsnglvYQvSrJeoFR70mCQ",
"version": {
"number": "7.7.0",
"build_flavor": "default",
"build_type": "tar",
"build_hash": "81a1e9eda8e6183f5237786246f6dced26a10eaf",
"build_date": "2020-05-16T02:01:37.602180Z",
"build_snapshot": false,
"lucene_version": "8.5.1",
"minimum_wire_compatibility_version": "6.8.0",
"minimum_index_compatibility_version": "6.0.0-beta1"
},
"tagline": "You Know, for Search"
}

看到类似以上结果说明es启动成功。


kibana安装配置启动

下载

1
2
3
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.7.0-linux-x86_64.tar.gz 
tar -zxf kibana-7.7.0-linux-x86_64.tar.gz
cd kibana-7.7.0-linux-x86_64

配置

修改 kibana-7.7.0-darwin-x86_64/config/kibana.yml

1
2
3
4
5
6
7
8
9
10
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "localhost"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://localhost:9200"]

启动

1
2
cd kibana-7.7.0-linux-x86_64/
./bin/kinana

后台运行可以用 ./bin/kinana &

验证是否启动成功

浏览器里输入 http://127.0.0.1:5601/ 如果可以打开页面说明启动成功


logstash安装配置启动

下载

1
2
3
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.7.0.tar.gz
tar -zxf logstash-7.7.0.tar.gz
cd logstash-7.7.0

配置

在 config 目录下新建配置 logstash-wkq-test.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
# Sample Logstash configuration for creating a simple
# file -> Logstash -> Elasticsearch pipeline.

input {
file {
path => [ "/Users/weikeqin1/WorkSpaces/java/springboot-test/logs/entrance.log", "/Users/weikeqin1/WorkSpaces/java/springboot-test/logs/*.log" ]
type => "entrance"
start_position => "beginning"
}

file {
path => [ "/Users/weikeqin1/WorkSpaces/java/springboot-test/logs/error*.log" ]
type => "error"
start_position => "beginning"
}

file {
path => "/var/log/apache/access.log"
type => "apache"
start_position => "beginning"
}

}


filter {
if [path] =~ "entrance" {
mutate { replace => { "type" => "entrance_log" } }
grok {
match => { "msg" => "%{COMBINEDAPACHELOG}" }
}
} else if [path] =~ "error" {
mutate { replace => { type => "error_log" } }
}

date {
match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss" ]
}

}


output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "log-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}

配置文件中的 grok 对应解析日志的字段
日志样例

1
2020-05-17 11:14:22.887 INFO [XNIO-1 task-9] cn.wkq.controller.TestController.test|{"_traceId":"1589685262886CY74","_method":"test","param":{"appKey":"wkq"}}

解析规则

1
2
3
%{DATA:logDate} %{DATA:logTime} %{DATA:logLevel} \[%{DATA:thread}\]%{DATA:classnameMethod}\|\{%{DATA:msg}\} 

%{DATA:logDate} %{DATA:logTime} %{DATA:logLevel} \[%{DATA:thread}\] %{DATA:classnameMethod}\|\{\"_traceId\":\"%{DATA:traceId}\"\,\"_method\":\"%{DATA:method}\",\"param\"\:\{%{DATA:param}\}

可以在kibana http://127.0.0.1:5601/app/kibana#/dev_tools/grokdebugger 调试

启动

1
./bin/logstash -f ./config/logstash-wkq-test.conf

测试

用压测工具压测程序,会生成很多日志,压测的时候看监控,程序才占用75M内存,logstash就占1000m内存,实际使用500m左右,elasticsearch用的默认配置,也占1000m内存,实际使用750m左右
logstash太占内存了

filebeat

下载安装

1
2
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.0-linux-x86_64.tar.gz
tar xzvf filebeat-7.7.0-linux-x86_64.tar.gz

配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

# Change to true to enable this input configuration.
enabled: true

# Paths that should be crawled and fetched. Glob based paths.
paths:
- /Users/weikeqin1/WorkSpaces/java/springboot-test/logs/*.log
#- c:\programdata\elasticsearch\logs\*

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]

# Protocol - either `http` (default) or `https`.
#protocol: "https"

# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]

# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"

启动

1
2
cd filebeat-7.7.0-linux-x86_64
sudo ./filebeat -e

在kibana查看日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
{
"_index" : "filebeat-7.7.0-2020.05.17",
"_type" : "_doc",
"_id" : "y42gIHIBU4Je81Q8Jaun",
"_score" : null,
"_source" : {
"log" : {
"file" : {
"path" : "/Users/weikeqin1/WorkSpaces/java/springboot-test/logs/springboot-test.2020-05-17_0.log"
},
"offset" : 371432
},
"tags" : [
"beats_input_codec_plain_applied"
],
"host" : {
"architecture" : "x86_64",
"id" : "DAA105AB-C93E-5E37-AFBF-A747032048AF",
"ip" : [
"xx::xx:xx:xx:xx",
"192.168.0.110",
"xx::xx:xx:xx:xx",
"xx::xx:xx:xx:xx"
],
"name" : "192.168.0.110",
"mac" : [
"xx:xx:xx:xx:xx:xx",
"xx:xx:xx:xx:xx:xx",
"xx:xx:xx:xx:xx:xx",
"xx:xx:xx:xx:xx:xx",
"xx:xx:xx:xx:xx:xx",
"xx:xx:xx:xx:xx:xx"
],
"os" : {
"version" : "10.13.6",
"build" : "17G8030",
"kernel" : "17.7.0",
"name" : "Mac OS X",
"family" : "darwin",
"platform" : "darwin"
},
"hostname" : "192.168.0.110"
},
"ecs" : {
"version" : "1.5.0"
},
"agent" : {
"id" : "1dcf526f-dc97-4a59-bccc-4751954962e6",
"type" : "filebeat",
"version" : "7.7.0",
"ephemeral_id" : "810f6da6-b38a-4071-8d2f-9c45dee14440",
"hostname" : "192.168.0.110"
},
"@version" : "1",
"@timestamp" : "2020-05-17T03:14:23.865Z",
"input" : {
"type" : "log"
},
"message" : """2020-05-17 11:14:22.879 [http-nio-11011-exec-20] INFO [cn.wkq.controller.TestController] [32] - {"_traceId":"15896852628794XZ3","_method":"TestController#test#1589685262879#"}"""
},
"sort" : [
1589685263865
]
}

References

[1] elastic官网
[2] elasticsearch官网中文简介
[3] elasticsearch官网logstash中文简介
[4] elasticsearch官网kibana中文简介
[5] elasticsearch下载安装启动及配置
[6] logstash-configuration
[7] kibana-getting-started
[8] beats-getting-started
[9] logstash-patterns