1. Building the Environment
172.19.2.51: elasticsearch+kibana+logstash+kopf 172.19.2.50: elasticsearch+nginx+filebeat 172.19.2.49: elasticsearch
The access log of nginx is the content we want to collect, which is transmitted by filebeat, so neither nginx nor filebeat runs in docker.
All other components run in docker, version 5
II. 172.19.2.51 Installation of elk components
1. Install docker-compose
curl -L https://github.com/docker/compose/releases/download/1.3.1/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose vim /etc/profile export PATH="$PATH:/usr/local/bin" source /etc/profile echo $PATH
2. Adjust the number of virtual memory in a single process. If you do not adjust the startup container, you will report an error.
sysctl -a | grep vm.max_map_count sysctl -w vm.max_map_count=262144
3. Create configuration file directories and files
Create an elastic search data storage directory
mkdir -pv /root/elk/elasticsearch
Create the elastic search configuration file directory
mkdir -pv /root/elk/es
Create kibana configuration file directory
mkdir -pv /root/elk/kibana
Create logstash configuration file directory
mkdir -pv /root/elk/logstash
Create the elastic search configuration file
vim /root/elk/es/elasticsearch.yml network.bind_host: 0.0.0.0 network.host: 172.19.2.51 cluster.name: es-cluster node.name: "es-node1" node.master: true discovery.zen.minimum_master_nodes: 1 discovery.zen.ping.unicast.hosts: - 172.19.2.51 - 172.19.2.50 - 172.19.2.49
Create kibana configuration file
vim /root/elk/kibana/kibana.yml port: 5601 host: "0.0.0.0" elasticsearch_url: "http://172.19.2.50:9100" elasticsearch_preserve_host: true kibana_index: ".kibana" default_app_id: "discover" request_timeout: 300000 shard_timeout: 0 verify_ssl: true bundled_plugin_ids: - plugins/dashboard/index - plugins/discover/index - plugins/doc/index - plugins/kibana/index - plugins/markdown_vis/index - plugins/metric_vis/index - plugins/settings/index - plugins/table_vis/index - plugins/vis_types/index - plugins/visualize/index
Create logstash configuration file
vim /root/elk/logstash/logstash.conf
input {
beats {
port => 20000
codec => "json"
}
}
output {
elasticsearch {
hosts => "172.19.2.50:9100"
index => "nginx" }
}Create docker-compose configuration file
vim /root/elk/docker-compose.yml elasticsearch: image: elasticsearch:5 command: elasticsearch environment: - "ES_JAVA_OPTS=-Xmx1g -Xms1g" volumes: - ./elasticsearch:/usr/share/elasticsearch/data - ./es/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml ports: - "9200:9200" - "9300:9300" logstash: image: logstash:latest command: logstash -w 4 -f /etc/logstash/conf.d/logstash.conf environment: - LS_HEAP_SIZE=2048m volumes: - ./logstash/logstash.conf:/etc/logstash/conf.d/logstash.conf ports: - "20000:20000" kibana: image: kibana:latest volumes: - ./kibana/kibana.yml:/etc/kibana/kibana.yml ports: - "5601:5601" kopf: image: lmenezes/elasticsearch-kopf ports: - "80:80" environment: - KOPF_SERVER_NAME=kopf - KOPF_ES_SERVERS=172.19.2.50:9100
4. Start docker-compose
cd /root/elk docker-compose up docker-compose ps
3. 172.19.2.51 Install elastic search and nginx+filebeat
1. Install docker-compose
curl -L https://github.com/docker/compose/releases/download/1.3.1/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose vim /etc/profile export PATH="$PATH:/usr/local/bin" source /etc/profile echo $PATH
2. Adjusting the number of virtual memory in a single process
sysctl -a | grep vm.max_map_count sysctl -w vm.max_map_count=262144
3. Create configuration file directories and files
Create an elastic search data storage directory
mkdir -pv /root/elk/elasticsearch
Create the elastic search configuration file directory
mkdir -pv /root/elk/es
Create the elastic search configuration file
vim /root/elk/es/elasticsearch.yml network.bind_host: 0.0.0.0 network.host: 172.19.2.50 cluster.name: es-cluster node.name: "es-node2" node.master: true discovery.zen.minimum_master_nodes: 1 discovery.zen.ping.unicast.hosts: - 172.19.2.51 - 172.19.2.50 - 172.19.2.49
Create docker-compose configuration file
vim /root/elk/docker-compose.yml elasticsearch: image: elasticsearch:5 command: elasticsearch environment: - "ES_JAVA_OPTS=-Xmx1g -Xms1g" volumes: - ./elasticsearch:/usr/share/elasticsearch/data - ./es/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml ports: - "9200:9200" - "9300:9300"
Modify the nginx configuration file (this nginx is used to bring back port 9200 to 9100 of the elastic search cluster, that is, the 9200 ports of the three hosts in the es cluster are accessed through 172.19.2.50:9200, and we collect the 80 port access log of this nginx)
vim /etc/nginx/nginx.conf
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format logstash_json '{ "@timestamp": "$time_local", '
'"@fields": { '
'"remote_addr": "$remote_addr", '
'"remote_user": "$remote_user", '
'"body_bytes_sent": "$body_bytes_sent", '
'"request_time": "$request_time", '
'"status": "$status", '
'"request": "$request", '
'"request_method": "$request_method", '
'"http_referrer": "$http_referer", '
'"body_bytes_sent":"$body_bytes_sent", '
'"http_x_forwarded_for": "$http_x_forwarded_for", '
'"http_user_agent": "$http_user_agent" } }';
access_log /var/log/nginx/access.log logstash_json;
sendfile on;
keepalive_timeout 65;
upstream els {
server 172.19.2.49:9200 weight=1 max_fails=2 fail_timeout=1;
server 172.19.2.50:9200 weight=1 max_fails=2 fail_timeout=1;
server 172.19.2.51:9200 weight=1 max_fails=2 fail_timeout=1;
}
server {
listen 9100;
access_log /var/log/nginx/accessels.log logstash_json;
location / {
proxy_pass http://els/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
include /etc/nginx/conf.d/*.conf;
}4. Installing and configuring filebeat
cd /root/ curl -L -O https://download.elastic.co/beats/filebeat/filebeat-1.3.0-x86_64.rpm rpm -vi filebeat-1.3.0-x86_64.rpm vim /etc/filebeat/filebeat.yml filebeat: prospectors: - paths: - /var/log/nginx/access.log input_type: log multiline: negate: true match: after tail_files: false registry_file: /var/lib/filebeat/registry output: logstash: hosts: ["172.19.2.51:20000"] worker: 4 shipper: logging: files: rotateeverybytes: 10485760 # = 10MB
5. Start docker-compose, nginx, filebeat
cd /root/elk docker-compose up service nginx start service filebeat start
IV. 172.19.2.49 Install the elastic search node
1. Install docker-compose
curl -L https://github.com/docker/compose/releases/download/1.3.1/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose vim /etc/profile export PATH="$PATH:/usr/local/bin" source /etc/profile echo $PATH
2. Adjusting the number of virtual memory in a single process
sysctl -a | grep vm.max_map_count sysctl -w vm.max_map_count=262144
3. Create configuration file directories and files
Create an elastic search data storage directory
mkdir -pv /root/elk/elasticsearch
Create the elastic search configuration file directory
mkdir -pv /root/elk/es
Create the elastic search configuration file
vim /root/elk/es/elasticsearch.yml network.bind_host: 0.0.0.0 network.host: 172.19.2.49 cluster.name: es-cluster node.name: "es-node3" node.master: true discovery.zen.minimum_master_nodes: 1 discovery.zen.ping.unicast.hosts: - 172.19.2.51 - 172.19.2.50 - 172.19.2.49
Create docker-compose configuration file
vim /root/elk/docker-compose.yml elasticsearch: image: elasticsearch:5 command: elasticsearch environment: - "ES_JAVA_OPTS=-Xmx1g -Xms1g" volumes: - ./elasticsearch:/usr/share/elasticsearch/data - ./es/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml ports: - "9200:9200" - "9300:9300"
4. Start docker-compose
cd /root/elk docker-compose up
5. ELK Plug-in Access Address
1,kopf
http://172.19.2.51/#!/cluster
2,kibana
http://172.19.2.51:5601/
3. All configuration files have been uploaded to git
https://github.com/xsllqs/Blogfile/tree/master/elk