概述
ELK是 elasticsearch + logstash + kibana 的简称。这套组合类似于MVC模型，其中logstash是controller层，数据首先传给它，它负责将数据进行过滤和格式化；elasticsearch是model层，负责数据的存储，建立搜索索引；kibana是view层，负责数据的展示。
所以使用中会涉及到以下几个知识 O(∩_∩)O~
elasticsearch
logstash
当然，如果熟悉ruby就更开心了：）
好了，说了一大堆，开始干活吧 : D
环境
系统：centos 6.x
软件版本：
elasticsearch-5.6.0
kibana-5.6.0
logstash-5.6.0
下载到本地:
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.0.rpm
wget https://artifacts.elastic.co/downloads/logstash/logstash-5.6.0.rpm
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.6.0-x86_64.rpm
安装软件
直接rpm安装
rpm -ivh elasticsearch-5.6.0.rpm
rpm -ivh logstash-5.6.0.rpm
rpm -ivh kibana-5.6.0-x86_64.rpm
配置elasticsearch
主要配置文件/etc/elasticsearch/elasticsearch.yml，添加如下内容
cluster.name: erp #指定个集群名字，等下会用到
node.name: node0 #节点名称也指定下
path.data: /opt/data #指定数据的存放路径
path.logs: /opt/data/logs #指定日志的存放路径，好排查问题
network.host: 172.16.93.237 #指定监听地址
http.port: 9200 #指定端口
如果没data目录，则新建下
mkdir -p /opt/data/logs
chown -R elasticsearch:elasticsearch /opt/data
启动elasticsearch服务
service elasticsearch start
查看运行状态
service elasticsearch status
curl http://ip:9200 如果有类似如下的输出则是正常启动了
# curl http://172.16.93.237:9200
{
"name" : "node0","cluster_name" : "erp","cluster_uuid" : "Vy0zvSCRQ-y_nAo9YHHRMQ","version" : {
"number" : "5.6.0","build_hash" : "781a835","build_date" : "2017-09-07T03:09:58.087Z","build_snapshot" : false,"lucene_version" : "6.6.0"
},"tagline" : "You Know,for Search"
}
max file descriptors [65535] for elasticsearch process is too low,increase to at least [65536]
elasticsearch对系统的文件打开数等有要求，需要支持下
vi /etc/security/limits.conf
添加或修改为如下内容:
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
vi /etc/security/limits.d/90-nproc.conf
添加或修改为如下内容:
* soft nproc 2048
错误2
system call filters Failed to install; check the logs and fix your configuration or disable system call filters at your own risk
说的好清楚了，关闭system_call_filter
vi /etc/elasticsearch/elasticsearch.yml
添加
bootstrap.system_call_filter: false
错误3
memory locking request for elasticsearch process but memory is not locked
尝试关闭bootstrap.memory_lock试试
vi /etc/elasticsearch/elasticsearch.yml
添加
bootstrap.memory_lock: false
错误4
kernel: Out of memory: Kill process
#表现为elasticsearch运行一会儿自动关闭了
是内存超了，如果你的系统内存不够用，可以限制下elasticsearch的内存开销(默认是2g)
vim /etc/elasticsearch/jvm.options
-Xms2g -Xmx2g 把这两个数值改小一点,比如
-Xms1g -Xmx1g
配置logstash
默认没有init.d的脚本，需要执行如下命令生成下
/usr/share/logstash/bin/system-install /etc/logstash/startup.options sysv
logstash的配置文件分三部分：input，filter，output。顾名思义，分别就是输入，过滤，输出了。具体的配置语法可以参考文档。
配置文件放置在/etc/logstash/conf.d/目录下。
在配置时，需结合日志格式来配置。比如用ELK来分析Nginx日志，Nginx的日志配置如下：
log_format main "$http_x_forwarded_for | $time_local | $request | $status | $body_bytes_sent | "
"$request_body | $content_length | $http_referer | $http_user_agent | nuid | "
"$http_cookie | $remote_addr | $hostname | $upstream_addr | $upstream_response_time | $request_time";
则可以在/etc/logstash/conf.d/目录下新建个Nginx-access.conf的配置文件，内容如下：
input {
file {
path => "/data/log/Nginx/*" #Nginx日志的路径 start_position => "beginning" sincedb_path => "/data/log/el/Nginx_progress" } } filter { ruby { init => "@kname = ['http_x_forwarded_for','time_local','request','status','body_bytes_sent','request_body','content_length','http_referer','http_user_agent','nuid','http_cookie','remote_addr','hostname','upstream_addr','upstream_response_time','request_time']" code => "
new_event = LogStash::Event.new(Hash[@kname.zip(event.get('message').split('|'))])
new_event.remove('@timestamp')
event.append(new_event)
" } if [request] { ruby { init => "@kname = ['method','uri','verb']" code => "
new_event = LogStash::Event.new(Hash[@kname.zip(event.get('request').split(' '))])
new_event.remove('@timestamp')
event.append(new_event)
" } if [uri] { ruby { init => "@kname = ['url_path','url_args']" code => "
new_event = LogStash::Event.new(Hash[@kname.zip(event.get('uri').split('?'))])
new_event.remove('@timestamp')
event.append(new_event)
" } kv { prefix => "url_" source => "url_args" field_split => "& " remove_field => [ "url_args","uri","request" ] } } } mutate { convert => [ "body_bytes_sent","integer","content_length","upstream_response_time","float","request_time","float" ] } date { match => [ "time_local","dd/MMM/yyyy:hh:mm:ss Z" ] locale => "en" } } output { elasticsearch { action => "index" hosts => "172.16.93.237:9200" index => "erp" } }
logstash的配置很丰富，比如我们还可以用json格式的，示例：
Nginx配置如下：
log_format main '{"data":"$time_local","ip":"$remote_addr","status":"$status","http_referer":"$http_referer","request_length":"$request_length","request":"$request","request_time":"$request_time","Authorization_User":"$http_authorization_user","http_user_agent":"$http_user_agent"}';
logstash的配置如下：
input {
file {
path => "/data/log/Nginx/*"
start_position => "beginning"
sincedb_path => "/data/log/el/Nginx_progress"
codec => "json"
}
}
output {
elasticsearch {
action => "index"
hosts => "172.16.93.237:9200"
index => "erp"
}
}
启动logstash
service logstash start
配置kibana
主要是配置下端口和地址
vim /etc/kibana/kibana.yml
配置或添加如下参数
server.port: 5601
server.host: "172.16.93.237"
server.name: "172.16.93.237"
elasticsearch.url: "http://172.16.93.237:9200"
启动kibana
service kibana start
如果一切正常，浏览器访问http://172.16.93.237:5601即可进入kibana
(PS：kibana默认是没用户名密码的，如果kibana暴露在公网，担心安全问题，可以通过Nginx等加一层验证。)
第一次进入需选择一个es的index，这里填入刚才配置集群erp*,点击create即可。之后就愉快地使用吧。
结语
ELK解决日志分析及可视化的问题，它的精华在于logstash，这也是要多花点时间的地方。
当前的场景是基于一台日志服务器搭建的，如果要采集多台服务器的日志时，可使用filebeat + logstash + elasticsearch + kibana 的架构。
参考
https://kibana.logstash.es/content/logstash/examples/nginx-access.html
https://my.oschina.net/shawnplaying/blog/670217
http://www.jb51.cc/article/p-olvfuvyc-boc.html
https://es.xiaoleilu.com/010_Intro/05_What_is_it.html
https://www.gitbook.com/book/chenryn/elk-stack-guide-cn/details
