My ELK is built with Docker!

I Instructions before installation

  1. The following steps are operated in centos 7 in VMware. The ip address is;

    Note that it's best to unify the version when installing, otherwise many problems will appear later. Search the official website for the corresponding image and check the version under the Tags tag. At present, my latest Tags is 7.12.1, so the version number is uniformly added when pulling the image. Official website image address

  2. Before setting up an ELK, Docker needs to be familiar with the relevant instructions of Docker, such as copying container file docker cp, forcibly deleting container: docker rm -f container id, creating network: docker network create elk, viewing logs: docker logs container, etc.

  3. To facilitate the attachment of subsequent files, first create the following directory / usr/local/elk, and then execute mkdir /usr/local/elk/{elasticsearch,kibana,logstash} to create three corresponding directories. Therefore, the following operations are performed under / usr/local/elk unless otherwise specified.

  4. In order to communicate between containers, you need to create a network with docker: docker network create elk.

II Install Docker

You can install docker step by step according to the official website. It's very simple.

Install Docker Engine on CentOS

After installation, you can see the following version: docker -v
View docker details: docker info

III Docker installation ElasticSearch

1. Search, download and view images

# Search image
docker search elasticsearch
# Pull the image of version 7.12.1
docker pull elasticsearch:7.12.1
# View all mirrors
docker images

2. Copy the configuration file

# Run elasticsearch
docker run -d --name es --net elk -P -e "discovery.type=single-node" elasticsearch:7.12.1
# Enter the container to view the configuration file path
docker exec -it es /bin/bash
cd config

In config, you can see elasticsearch YML configuration file, and then execute pwd. You can see that the current directory is: / usr/share/elasticsearch/config, so exit the container and execute the copy of the file:

# Copy the / search / elask configuration file into the / elastr / local container
docker cp es:/usr/share/elasticsearch/config/elasticsearch.yml elasticsearch/

# Modify file permissions
chmod 666 elasticsearch/elasticsearch.yml

# Create the data directory under the elasticsearch directory and modify the permissions at the same time
chmod -R 777 elasticsearch/data

Note: the permission of the file to be modified here is writable. Otherwise, after mounting, modify the configuration file externally, and the configuration file inside the container will not be changed. At the same time, create a data directory for mounting.

3. Re run the container and mount:

# Delete the old container first
docker rm -f es

# Run new container
docker run -d --name es \
--net elk \
-p 9200:9200 -p 9300:9300 \
-e "discovery.type=single-node" \
--privileged=true \
-v $PWD/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
-v $PWD/elasticsearch/data/:/usr/share/elasticsearch/data \


  • -p (lowercase) mapping port number, host port: container port
  • -P (uppercase) specifies the port number for the container at random
  • -v mount the container
  • – name specifies the container alias
  • – net connection to the specified network
  • -e specifies the environment variable when starting the container
  • -d background running container
  • – privileged=true enables the container to mount directories

4. View container

# View containers in operation
docker ps

# View container log
docker logs es

curl localhost:9200

If the log is correct, you can use curl ip:9200 to view the results, or you can use the browser to enter http://ip:9200 see. At this point, the elasticsearch installation is complete.

At the same time, we can see the data in the container in the data directory, and the modifications in the host will also be synchronized to the interior of the container.

IV Docker installation elasticsearch head (optional)

Note: This is the visual interface of ES. It is convenient to check whether the log has been successfully entered into es during later debugging. For items that do not have to be installed, you can also skip this step.

1. Pull the image

docker pull mobz/elasticsearch-head:5

2. Operation container

docker run -d --name es_admin --net elk -p 9100:9100 mobz/elasticsearch-head:5
Note: here -- net elk is used, that is, the network created between us;
If it is not created, you need to modify the configuration accessories in the next step. If it is created as before, you can skip it directly.

3. Edit profile (optional)

#Copy the configuration file to the current directory
docker cp es_admin:/usr/src/app/Gruntfile.js ./

Edit the configuration file and add: hostname: '' under connect - > server - > options

#Copy the file back
docker cp Gruntfile.js es_admin:/usr/src/add/
# Restart container
docker restart es_admin

Because vi and vim are not installed inside the image, it cannot be edited directly inside the container. We need to copy it and overwrite it after modification.
After restarting the container, pass http://ip:9100 Access to query the data in ES.

4. There is no data problem on the page

If the nodes and indexes are completely displayed after opening the page, but no data is displayed in the data browsing, we need to change a configuration file because the strict verification of the request header is added after ES 6 (we installed version 7.12.1):

docker cp es_admin:/usr/src/app/_site/vendor.js ./
vi vendor.js
docker cp vendor.js es_admin:/usr/src/app/_site/

There are a lot of contents in this configuration file. Here you need to understand some operations of vim:
Press ESC to enter the command mode, then: set nu displays the line number, and then: the line number jumps to the corresponding line.
Here we need to modify: (note that the whole line is not replaced)
Line 6886: contentType: "application/json;charset=UTF-8"
Line 7573: contentType === "application/json;charset=UTF-8"
After the change, copy the configuration file back to the container. You can refresh the page directly without restarting.

V Docker installation Kibana

The steps are roughly the same as elasticsearch:

1. Pull the image

docekr search kibana
docker pull kibana: 7.12.1

# Start the kibana container and connect to the same network
docker run -d --name kibana --net elk -P -e "ELASTICSEARCH_HOSTS=http://es:9200" -e "I18N_LOCALE=zh-CN" kibana:7.12.1

Note: - e "elasticsearch_hosts"= http://es:9200 "Indicates the connection to the elasticsearch container just started, because in the same network (elk), the address can be directly filled in the container name + port, that is, es: 9200 or , i.e http://ip: Port.

2. Copy files

docker cp kibana:/usr/share/kibana/config/kibana.yml kibana/

chmod 666 kibana/kibana.yml

After copying, modify the configuration file, mainly elastissearch Hosts and add I18N Locale configuration:

  1. The es address is changed to the es address just installed. Because of the isolation of the container, it's best to fill in here http://ip:9200 ;
  2. kibana interface is in English by default, and I18N can be added to the configuration file Locale: zh CN (note that there is a space after the colon).

In this way, with the configuration file, you don't have to specify the environment variable through - e when starting the container.


# Default Kibana configuration for docker target kibana "0"
# elasticsearch address
elasticsearch.hosts: [ "" ]
monitoring.ui.container.elasticsearch.enabled: true
# Open the Sinicization of kibana
i18n.locale: zh-CN

Note: if you start by mounting a configuration file, elasticsearch Hosts this needs to be filled in http://ip:9200 , instead of using the container name, otherwise kibana connection es will fail.

3. Reopen a container

#Delete the original unmounted container
docker rm -f kibana

# Start the container and mount it
docker run -d --name kibana \
-p 5601:5601 \
-v $PWD/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml \
--net elk \

4. View results

Open the browser and enter: http://ip:5601 Open kibana console. If it fails to open successfully, you can use: docker logs kibana to check the container log to see if it runs incorrectly, etc.

Please refer to the official website: Install Kibana with Docker

Vi Docker installation LogStash

Continue to install Logtash as above:

1. Pull the image and copy the configuration

docker pull logstash:7.12.1

docker run -d -P --name logstash --net elk logstash:7.12.1

# Copy data
docker cp logstash:/usr/share/logstash/config logstash/
docker cp logstash:/usr/share/logstash/data logstash/
docker cp logstash:/usr/share/logstash/pipeline logstash/

#Folder empowerment
chmod -R 777 logstash/

2. Modify the corresponding configuration file

  • Modify logstash.config under logstash/config YML file, mainly modify the address of es: ""
xpack.monitoring.elasticsearch.hosts: [ "" ]
  • Modify logstash. Under logstash/pipeline Conf file:
input {
  tcp {
    mode => "server"
    host => ""  # Allow any host to send logs
    port => 5044
    codec => json_lines    # data format

output {
  elasticsearch {
      hosts  => [" "] # elasticsearch address and port
      index  => "elk"         # Specify index name
      codec  => "json"
  stdout {
    codec => rubydebug

3. Start the container and mount it

#Note to delete the previous container first
docker rm -f logstash

# Mount and mount the container
docker run -d --name logstash --net elk \
--privileged=true \
-p 5044:5044 -p 9600:9600 \
-v $PWD/logstash/data/:/usr/share/logstash/data \
-v $PWD/logstash/config/:/usr/share/logstash/config \
-v $PWD/logstash/pipeline/:/usr/share/logstash/pipeline \

4. View log

docker logs -f logstash

Every time you start the container, you'd better check the log to facilitate the subsequent steps. If there is a problem, you can see which problem is and solve it easily.

VII Create springboot application

This is relatively simple, mainly including several configuration files:

  1. pom.xml file, introducing the dependency of logstash:
  1. Create a log / logback spring XML file under resource, which we mainly fill in here
    < destination > IP: Port < / destination >, and the < pattern > label can be modified according to personal usage.
<?xml version="1.0" encoding="UTF-8"?>
    <include resource="org/springframework/boot/logging/logback/base.xml" />
    <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <!-- Log output code -->
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder">
                        <!--es Index name -->
                        <!--apply name -->
                        <!--Print time -->
                        "timestamp": "%d{yyyy-MM-dd HH:mm:ss.SSS}",
                        <!--Thread name -->
                        "thread": "%thread",
                        <!--log level -->
                        "level": "%level",
                        <!--Log name -->
                        "logger_name": "%logger",
                        <!--log information -->
                        "message": "%msg",
                        <!--Log stack -->
                        "stack_trace": "%exception"
    <root level="INFO">
        <appender-ref ref="LOGSTASH" />
        <appender-ref ref="CONSOLE" />
  1. Finally, modify the application YML file:
  config: classpath:log/logback-spring.xml

After startup, you can see the output log in the elastic search head interface:

At the same time, you can see the index elk added on kibana page:

So far, the process of building an elk with Docker has been completed.

VII Postscript

During the construction of ELK, the following problems are encountered, which are hereby recorded:

1. Insufficient docker pull memory

Since I built it on centos 7 installed by VMware, the memory attached to the root directory is small when installing the system. Therefore, when docker pulls the image, it will prompt that there is insufficient memory at the Logstash step and I don't want to reinstall the system. Therefore, you can change the default directory of container volume download by editing docker's configuration file.

# View the default download address of docker
docker info | grep "Docker Root Dir"

# Create directory
mkdir /home/docker-root

# Edit profile 
vi /usr/lib/systemd/system/docker.service

Editing docker In the Service configuration file, find ExecStart under the [Service] node and add: - graph = / home / docker root after it. Docker root is the directory just created:

ExecStart=/usr/bin/dockerd --graph=/home/docker-root

Finally, restart docker:

systemctl daemon-reload
systemctl restart docker
systemctl enable docker

2.Logstash port mapping

Edit logstash When using the conf file, it should be noted that the exposed ports also need container mapping, otherwise the springboot will not be connected to logstash, such as:

input {
  tcp {
    mode => "server"
    host => ""  # Allow any host to send logs
    port => 5044
    codec => json_lines    # data format

Here, any host is allowed to send logs through port 5044. When starting the container, you need -p 5044:5044 to map out the port at the same time, otherwise the springboot will not be connected to logstash during startup.

3. Insufficient memory when starting the container

When starting elasticsearch or logstash container, there may be insufficient memory. We can modify the memory allocation by editing and modifying the specified file:

# Find JVM Options file
find / -name jvm.options

Find the corresponding JVM Options modifies - Xms (minimum memory) and - Xmx (maximum memory) in the file.

Or when you start the elasticsearch container, you can directly add the environment variables as shown below, but the logstash is not found at present. You can modify the JVM Options to modify.

-e "ES_JAVA_OPTS=-Xms1g -Xmx1g" 

Keywords: Docker ELK

Added by foreknowapparel on Wed, 09 Feb 2022 10:01:42 +0200