Elasticsearch Logstash Kibana 笔记
Elastic Stack 是适用于数据采集、充实、存储、分析和可视化的一组开源工具。人们通常将 Elastic Stack 称为 ELK Stack（代指 Elasticsearch、Logstash 和 Kibana）
Elasticsearch 是一个分布式的开源搜索和分析引擎，适用于所有类型的数据，包括文本、数字、地理空间、结构化和非结构化数据。 类似于MySQL，用来存储和分析数据。
Logstash可用来对数据进行聚合和处理，并将数据发送到 Elasticsearch。Logstash 是一个开源的服务器端数据处理管道，允许您在将数据索引到 Elasticsearch 之前同时从多个来源采集数据，并对数据进行充实和转换。 类似于程序，抽取解析数据并储存到ES
Kibana 是一款适用于 Elasticsearch 的数据可视化和管理工具。类似于使用MySQL时用的Navicat
简单的说，LogStash用来收取解析日志并发消息写入ES，Elasticsearch用来存储和分析查询，Kibana用来查看
es下载地址 https://www.elastic.co/cn/downloads/elasticsearch
logstash下载地址 https://www.elastic.co/cn/downloads/logstash
kibana下载地址 https://www.elastic.co/cn/downloads/kibana
filebeats
elasticsearch-7.x 要求JDK版本大于等于11
ZBMAC-C02PGMT0F:~ weikeqin1$ java -version
java version "14" 2020-03-17
Java(TM) SE Runtime Environment (build 14+36-1461)
Java HotSpot(TM) 64-Bit Server VM (build 14+36-1461, mixed mode, sharing)
ZBMAC-C02PGMT0F:~ weikeqin1$elasticsearch安装配置启动
下载
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.7.0-linux-x86_64.tar.gz
tar -xzf elasticsearch-7.7.0-linux-x86_64.tar.gz
cd elasticsearch-7.7.0/配置
修改
elasticsearch-7.7.0/config/elasticsearch.yml
action.destructive_requires_name: true
xpack.ml.enabled: true启动
cd elasticsearch-7.7.0/
./bin/elasticsearch后台启动可以用
./bin/elasticsearch -d
验证是否启动成功
在浏览器里输入 http://localhost:9200/?pretty 或者用
curl 'http://localhost:9200/?pretty'
{
"name": "192.168.0.110",
"cluster_name": "elasticsearch",
"cluster_uuid": "aHsnglvYQvSrJeoFR70mCQ",
"version": {
"number": "7.7.0",
"build_flavor": "default",
"build_type": "tar",
"build_hash": "81a1e9eda8e6183f5237786246f6dced26a10eaf",
"build_date": "2020-05-16T02:01:37.602180Z",
"build_snapshot": false,
"lucene_version": "8.5.1",
"minimum_wire_compatibility_version": "6.8.0",
"minimum_index_compatibility_version": "6.0.0-beta1"
},
"tagline": "You Know, for Search"
}看到类似以上结果说明es启动成功。
kibana安装配置启动
下载
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.7.0-linux-x86_64.tar.gz
tar -zxf kibana-7.7.0-linux-x86_64.tar.gz
cd kibana-7.7.0-linux-x86_64配置
修改
kibana-7.7.0-darwin-x86_64/config/kibana.yml
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "localhost"
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://localhost:9200"]
启动
cd kibana-7.7.0-linux-x86_64/
./bin/kinana后台运行可以用
./bin/kinana &
验证是否启动成功
浏览器里输入 http://127.0.0.1:5601/ 如果可以打开页面说明启动成功
logstash安装配置启动
下载
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.7.0.tar.gz
tar -zxf logstash-7.7.0.tar.gz
cd logstash-7.7.0配置
在 config 目录下新建配置
logstash-wkq-test.conf
# Sample Logstash configuration for creating a simple
# file -> Logstash -> Elasticsearch pipeline.
input {
file {
path => [ "/Users/weikeqin1/WorkSpaces/java/springboot-test/logs/entrance.log", "/Users/weikeqin1/WorkSpaces/java/springboot-test/logs/*.log" ]
type => "entrance"
start_position => "beginning"
}
file {
path => [ "/Users/weikeqin1/WorkSpaces/java/springboot-test/logs/error*.log" ]
type => "error"
start_position => "beginning"
}
file {
path => "/var/log/apache/access.log"
type => "apache"
start_position => "beginning"
}
}
filter {
if [path] =~ "entrance" {
mutate { replace => { "type" => "entrance_log" } }
grok {
match => { "msg" => "%{COMBINEDAPACHELOG}" }
}
} else if [path] =~ "error" {
mutate { replace => { type => "error_log" } }
}
date {
match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss" ]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "log-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
配置文件中的 grok 对应解析日志的字段
日志样例
2020-05-17 11:14:22.887 INFO [XNIO-1 task-9] cn.wkq.controller.TestController.test|{"_traceId":"1589685262886CY74","_method":"test","param":{"appKey":"wkq"}}解析规则
%{DATA:logDate} %{DATA:logTime} %{DATA:logLevel} \[%{DATA:thread}\]%{DATA:classnameMethod}\|\{%{DATA:msg}\}
%{DATA:logDate} %{DATA:logTime} %{DATA:logLevel} \[%{DATA:thread}\] %{DATA:classnameMethod}\|\{\"_traceId\":\"%{DATA:traceId}\"\,\"_method\":\"%{DATA:method}\",\"param\"\:\{%{DATA:param}\}可以在kibana http://127.0.0.1:5601/app/kibana#/dev_tools/grokdebugger 调试
启动
./bin/logstash -f ./config/logstash-wkq-test.conf测试
用压测工具压测程序，会生成很多日志，压测的时候看监控，程序才占用75M内存，logstash就占1000m内存，实际使用500m左右，elasticsearch用的默认配置，也占1000m内存，实际使用750m左右
logstash太占内存了
filebeat
下载安装
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.7.0-linux-x86_64.tar.gz
tar xzvf filebeat-7.7.0-linux-x86_64.tar.gz配置
#=========================== Filebeat inputs =============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /Users/weikeqin1/WorkSpaces/java/springboot-test/logs/*.log
#- c:\programdata\elasticsearch\logs\*
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
启动
cd filebeat-7.7.0-linux-x86_64
sudo ./filebeat -e在kibana查看日志
{
"_index" : "filebeat-7.7.0-2020.05.17",
"_type" : "_doc",
"_id" : "y42gIHIBU4Je81Q8Jaun",
"_score" : null,
"_source" : {
"log" : {
"file" : {
"path" : "/Users/weikeqin1/WorkSpaces/java/springboot-test/logs/springboot-test.2020-05-17_0.log"
},
"offset" : 371432
},
"tags" : [
"beats_input_codec_plain_applied"
],
"host" : {
"architecture" : "x86_64",
"id" : "DAA105AB-C93E-5E37-AFBF-A747032048AF",
"ip" : [
"xx::xx:xx:xx:xx",
"192.168.0.110",
"xx::xx:xx:xx:xx",
"xx::xx:xx:xx:xx"
],
"name" : "192.168.0.110",
"mac" : [
"xx:xx:xx:xx:xx:xx",
"xx:xx:xx:xx:xx:xx",
"xx:xx:xx:xx:xx:xx",
"xx:xx:xx:xx:xx:xx",
"xx:xx:xx:xx:xx:xx",
"xx:xx:xx:xx:xx:xx"
],
"os" : {
"version" : "10.13.6",
"build" : "17G8030",
"kernel" : "17.7.0",
"name" : "Mac OS X",
"family" : "darwin",
"platform" : "darwin"
},
"hostname" : "192.168.0.110"
},
"ecs" : {
"version" : "1.5.0"
},
"agent" : {
"id" : "1dcf526f-dc97-4a59-bccc-4751954962e6",
"type" : "filebeat",
"version" : "7.7.0",
"ephemeral_id" : "810f6da6-b38a-4071-8d2f-9c45dee14440",
"hostname" : "192.168.0.110"
},
"@version" : "1",
"@timestamp" : "2020-05-17T03:14:23.865Z",
"input" : {
"type" : "log"
},
"message" : """2020-05-17 11:14:22.879 [http-nio-11011-exec-20] INFO [cn.wkq.controller.TestController] [32] - {"_traceId":"15896852628794XZ3","_method":"TestController#test#1589685262879#"}"""
},
"sort" : [
1589685263865
]
}References
[1] elastic官网
[2] elasticsearch官网中文简介
[3] elasticsearch官网logstash中文简介
[4] elasticsearch官网kibana中文简介
[5] elasticsearch下载安装启动及配置
[6] logstash-configuration
[7] kibana-getting-started
[8] beats-getting-started
[9] logstash-patterns