Elasticsearch 是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。
Logstash 是一个完全开源的工具，他可以对你的日志进行收集、分析，并将其存储供以后使用（如，搜索）
kibana 也是一个开源和免费的工具，他Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。
环境：192.168.50.119：ELK+Nginx
192.168.50.120：Redis+Logstash
架构图
部署流程：
192.168.50.119 ELK服务器
1.安装JDK
Logstash的运行依赖于Java运行环境， Logstash 1.5以上版本不低于java 7推荐使用最新版本的Java，我这里使用了1.8版本
tar-zxfjdk-8u45-linux-x64.tar.gz-C/usr/local/ vim/etc/profile#设置环境变量 exportJAVA_HOME=/usr/local/jdk1.8.0_45 exportPATH=$PATH:$JAVA_HOME/bin exportCLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH source/etc/profile#使环境变量生效
验证是否安装成功
[root@localhost~]#java-version javaversion"1.8.0_45" Java(TM)SERuntimeEnvironment(build1.8.0_45-b14) JavaHotSpot(TM)64-BitServerVM(build25.45-b02,mixedmode)
2.安装Logstash（日志收集、分析，并将其存储供以后使用）
wgethttps://download.elastic.co/logstash/logstash/logstash-2.4.0.tar.gz tar�Czxflogstash-2.4.0.tar.gz-C/usr/local/
验证logstash是否安装成功
[root@localhost~]#/usr/local/logstash-2.4.0/bin/logstash-e'input{stdin{}}output{stdout{}}'
Settings:Defaultpipelineworkers:1
Logstashstartupcompleted
等待输入:helloworld
2016-11-28T20:32:07.853Zlocalhost.localdomainhelloworld
我们可以看到，我们输入什么内容logstash按照某种格式输出，其中-e参数参数允许Logstash直接通过命令行接受设置。
这点尤其快速的帮助我们反复的测试配置是否正确而不用写配置文件。使用CTRL-C命令可以退出之前运行的Logstash。
3.部署Nginx并收集日志
yum-yinstallNginx 设置Nginx的log格式 vim/etc/Nginx/Nginx.conf log_formatmain'$remote_addr-$remote_user[$time_local]"$request"' '$status$body_bytes_sent"$http_referer"' '"$http_user_agent"$http_x_forwarded_for$request_length$msec$connection_requests$request_time';
启动Nginx
serviceNginxstart
mkdir/usr/local/logstash-2.4.0/conf/#创建logstash配置目录 定义logstash配置文件，用来收集Nginx日志 [root@localhostconf]#catlogstash_Nginx.conf input{ file{ path=>["/var/log/Nginx/access.log"] type=>"Nginx_log" } } output{ redis{ host=>"192.168.50.120" key=>'logstash-redis' data_type=>'list' } stdout{ codec=>rubydebug } }
4.安装部署redis
192.168.50.120 服务器
yum-yinstallredis vim/etc/redis.conf bind192.168.50.120
启动
serviceredisstart
5.启动Logstash
[root@localhostconf]#/usr/local/logstash-2.4.0/bin/logstash-f./logstash_Nginx.conf--configtest#检查配置文件 ConfigurationOK
[root@localhostconf]#/usr/local/logstash-2.4.0/bin/logstashagent-f./logstash_Nginx.conf#将日志信息输出到redis服务器 Settings:Defaultpipelineworkers:1 Logstashstartupcompleted { "message"=>"192.168.50.114--[29/Nov/2016:00:58:43+0800]\"GET/HTTP/1.1\"3040\"-\"\"Mozilla/5.0(WindowsNT6.1;Win64;x64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/54.0.2840.99Safari/537.36\"\"-\"","@version"=>"1","@timestamp"=>"2016-11-28T18:55:49.587Z","path"=>"/var/log/Nginx/access.log","host"=>"localhost.localdomain","type"=>"Nginx_log" } { "message"=>"192.168.50.114--[29/Nov/2016:00:58:43+0800]\"GET/Nginx-logo.pngHTTP/1.1\"3040\"http://192.168.50.119/\"\"Mozilla/5.0(WindowsNT6.1;Win64;x64)AppleWebKit/537.36(KHTML,"@timestamp"=>"2016-11-28T18:55:49.590Z","type"=>"Nginx_log" } { "message"=>"192.168.50.114--[29/Nov/2016:00:58:43+0800]\"GET/poweredby.pngHTTP/1.1\"3040\"http://192.168.50.119/\"\"Mozilla/5.0(WindowsNT6.1;Win64;x64)AppleWebKit/537.36(KHTML,"type"=>"Nginx_log" }
6.安装部署Elasticsearch
192.168.50.119 ELK服务器
创建安装用户
groupaddelk useraddes-gelk
tar-xfelasticsearch-2.2.0.tar.gz-C/usr/local/ vim/usr/local/elasticsearch-2.2.0/config/elasticsearch.yml network.host:192.168.50.119#端口绑定ip地址 http.port:9200
启动
这里遇到一个坑：es用户默认是不能用root用户启动的。所以要切到普通用户启动
chown-Res.elk/usr/local/elasticsearch-2.2.0 su-es nohup/usr/local/elasticsearch-2.2.0/bin/elasticsearch>/usr/local/elasticsearch-2.2.0/nohub&
[root@localhostELK]#netstat-tunpl|grep9200 tcp00::ffff:192.168.50.119:9200:::*LISTEN2183/java
[root@localhostELK]#curlhttp://192.168.50.119:9200#查看状态
{
"name":"BloodBrothers","cluster_name":"elasticsearch","version":{
"number":"2.2.0","build_hash":"8ff36d139e16f8720f2947ef62c8167a888992fe","build_timestamp":"2016-01-27T13:32:39Z","build_snapshot":false,"lucene_version":"5.4.1"
},"tagline":"YouKnow,forSearch"
}
安装kopf和head插件
[root@localhostconf]#cd/usr/local/elasticsearch-2.2.0/bin/ [root@localhostbin]#./plugininstalllmenezes/elasticsearch-kopf ->Installinglmenezes/elasticsearch-kopf... Tryinghttps://github.com/lmenezes/elasticsearch-kopf/archive/master.zip... Downloading............................................................DONE Verifyinghttps://github.com/lmenezes/elasticsearch-kopf/archive/master.zipchecksumsifavailable... NOTE:Unabletoverifychecksumfordownloadedplugin(unabletofind.sha1or.md5filetoverify) Installedkopfinto/usr/local/elasticsearch-2.2.0/plugins/kopf
[root@localhostbin]#./plugininstallmobz/elasticsearch-head ->Installingmobz/elasticsearch-head... Tryinghttps://github.com/mobz/elasticsearch-head/archive/master.zip... Downloading.........................................................DONE NOTE:Unabletoverifychecksumfordownloadedplugin(unabletofind.sha1or.md5filetoverify) Installedheadinto/usr/local/elasticsearch-2.2.0/plugins/head
7.安装kibana
192.168.50.119 ELK服务器
安装
[root@localhostELK]#tar-xfkibana-4.4.0-linux-x64.tar.gz-C/usr/local/ [root@localhostELK]#cd/usr/local/kibana-4.4.0-linux-x64/
配置
[root@localhostkibana-4.4.0-linux-x64]#vimconfig/kibana.yml elasticsearch.url:"http://192.168.50.119:9200" server.port:5601 server.host:"0.0.0.0"
启动
[root@localhostkibana-4.4.0-linux-x64]#nohup/usr/local/kibana-4.4.0-linux-x64/bin/kibana>/usr/local/kibana-4.4.0-linux-x64/nohub.out&
[root@localhostELK]#netstat-tunpl|grep5601 tcp000.0.0.0:56010.0.0.0:*
浏览器访问http://192.168.50.119:5601/
8.安装logstash-server服务器
192.168.50.120 服务器
安装jdk和logstash
tar-zxfjdk-8u45-linux-x64.tar.gz-C/usr/local/ vim/etc/profile#设置环境变量 exportJAVA_HOME=/usr/local/jdk1.8.0_45 exportPATH=$PATH:$JAVA_HOME/bin exportCLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH source/etc/profile#使环境变量生效
tar�Czxflogstash-2.4.0.tar.gz-C/usr/local/ mkdir/usr/local/logstash-2.4.0/conf
将redis 中的数据发送到elasticsearch中
[root@localhostconf]#catlogstash_server.conf
input{
redis{
port=>"6379"
host=>"192.168.50.120"
data_type=>"list"
key=>"logstash-redis"
type=>"redis-input"
}
}
output{
elasticsearch{
hosts=>"192.168.50.119"
index=>"logstash-%{+YYYY.MM.dd}"
}
}
9.在Kibanda上创建Nginx日志监控视图
es常规操作
es健康状态 [root@localhost~]#curlhttp://192.168.50.119:9200/_cat/health?v epochtimestampclusterstatusnode.totalnode.datashardsprireloinitunassignpending_tasksmax_task_wait_timeactive_shards_percent 148034531523:01:55elasticsearchyellow11660060-50.0% health的状态包括：green,yellow,red. 列出节点 [root@localhost~]#curlhttp://192.168.50.119:9200/_cat/nodes?v hostipheap.percentram.percentloadnode.rolemastername 192.168.50.119192.168.50.1198990.00d*BloodBrothers 列出索引 [root@localhost~]#curlhttp://192.168.50.119:9200/_cat/indices?v healthstatusindexprirepdocs.countdocs.deletedstore.sizepri.store.size yellowopen.kibana11205.6kb5.6kb yellowopenlogstash-2016.11.2851104.9kb4.9kb
参考地址：
http://www.jb51.cc/article/p-ufvpepga-bkx.html http://www.jb51.cc/article/p-bgdyktgn-bhr.html
