ELK (Elasticsearch+logstash+kibana) detailed installation tutorial

1. Premise preparation

1.1 online installation jdk

1.1.1 is jdk installed

The operation of Elasticsearch requires a JDK environment, but the version above Elasticsearch 7 will bring its own JDK (the installation package has changed from tens of megabytes to hundreds of megabytes, which is why), and the version above Elasticsearch 7 needs jdk11. The jdk8 used in ordinary times does not meet the following requirements:

future versions of Elasticsearch will require Java 11; your Java version from [d:\Program Files\java\jdk1.8.0_91\jre] does not meet this requirement

1.1.2 installation jdk

Because you can ping www.baidu.com Com indicates that it is currently connected to the Internet. It is recommended to install jdk online directly, but it is not recommended after trying. The download speed is too slow. It is better to download locally and then upload and install.

Step 1: check the version of jdk in yum Library

yum search java|grep jdk

You can see that there are four versions: 1.6, 1.7, 1.8 and 11. To use version 11, I used version 1.8 before, and then prompted that the version is inconsistent (of course, there will be no error, but a warning)

Step 2: select java-11.0 installation

yum install java-11.0-openjdk*

Step 3: configure environment variables

vi /etc/profile

Here jdk1 The folder path of 8.0 is / usr / lib / JVM / java-1.8.0-openjdk-1.8.0.292 b10-1. el7_ 9.x86_ sixty-four

Add the following:

#set java environment
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64
JRE_HOME=$JAVA_HOME/jre
CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export JAVA_HOME JRE_HOME CLASS_PATH PATH

Note: Java above_ The version name corresponding to home may be different. Finally, find the corresponding path and change it.

Step 4: environment variables take effect

source /etc/profile

Step 5: check whether the jdk is installed successfully

java -version

1.2 download ELK locally

If linux is networked, ELK software can be downloaded directly through commands, but in most cases it is offline, and the download speed is very slow even in the networked state. It is far better to download the installation package locally and upload it to the server.

Three ELK software installation packages (Linux 64 bit, version 7.8.1) have been uploaded to Baidu cloud. Baidu cloud link: https://pan.baidu.com/s/1K4M0eBFwGsiiv3q3y-yaDQ Extraction code: bfp9

Download if necessary.

2. Install Elasticsearch

Brief introduction

Elasticsearch is a distributed, free and open source search and analysis engine for all types of data, including text, digital, geospatial, structured and unstructured data. Elasticsearch was developed on the basis of Apache Lucene and was first released by Elasticsearch N.V. (now elastic) in 2010. Elasticsearch is famous for its simple REST style API, distributed features, speed and scalability. It is the core component of elasticstack; Elastic Stack is a free and open source tool for data collection, expansion, storage, analysis and visualization. People usually call Elastic Stack ELK Stack (referring to elasticsearch, Logstash and Kibana). At present, Elastic Stack includes a series of lightweight data collection agents, which are collectively referred to as Beats and can be used to send data to elasticsearch.

For more information, please refer to:

Step 1: upload and unzip

Upload the installation package to the / opt directory, and execute in this directory:

 tar -zxvf elasticsearch-7.2.0-linux-86_64.tar.gz

After decompression, rename the directory:

mv elasticsearch-7.2.0 elasticsearch

Step 2: configure system parameters

The purpose of modifying system parameters is to ensure that the system has sufficient resources to start Elasticsearch.

(a) Set kernel parameters

vi /etc/sysctl.conf

Add the following parameters

vm.max_map_count=655360

Execute the following command to ensure that the parameters take effect:

sysctl -p

(b) Set resource parameters

vi /etc/security/limits.conf

Add the following at the end

* soft nofile 65536
* hard nofile 131072
* soft nproc 65536
* hard nproc 131072

(c) Set user resource parameters

vi /etc/security/limits.d/20-nproc.conf

Add the following:

# User settings elk
elk    soft    nproc     65536

Step 3: add new users

Because Elasticsearch does not allow root to be used as the program startup user, a new system user should be added, otherwise an error will be reported:

groupadd elk        #Create group elk

# Create the user es, belonging to the ES user group, with a password of 123456
useradd elk -g elk -p 123456

mkdir  -pv  /opt/elk/{data,logs} # Create data and log directories to prepare for the next step

# Modify file owner
chown -R elk:elk /opt/elk/
chown -R elk:elk /opt/elasticsearch/

Delete user name:

userdel  -r  user name

Step 4: Elasticsearch configuration

Modify the configuration file of Elasticsearch / opt / Elasticsearch / Elasticsearch yml. The configuration is as follows:

Note: when configuring parameters, there should be a space after the colon

Step 5: close the firewall

The firewall needs to be turned off for remote login. This option is optional. It doesn't need to be turned off.

# Turn off the firewall
systemctl stop firewalld

# Turn off firewall and start automatically
systemctl disable firewalld

Step 6: run Elasticsearch

Run command:

# Display execution
./opt/elasticsearch/bin/elasticsearch

# Background execution
./opt/elasticsearch/bin/elasticsearch -d

If the installation path is inconsistent, it needs to be changed.

Verify successful execution:

[root@localhost ~]# curl 'http://localhost:9200/?pretty'
{
  "name" : "O5BAVYE",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "rw1yjlzkSgODXkUVgIxmxg",
  "version" : {
    "number" : "6.4.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "e36acdb",
    "build_date" : "2018-09-13T22:18:07.696808Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
[root@localhost ~]#

If it returns success, the operation is successful.

Step 7: view the process

Elasticsearch cannot be queried by name, because elasticsearch runs in java and needs to be queried by port number:

netstat -tunlp|grep 9200

You can also query by:

ps aux|grep elasticsearch

Step 8: power on and self start

(a) Check the current startup service first

chkconfig --list

The startup services of the current system will be listed.

(b) System startup service file created

Go to CD / etc / init Execution directory, D:

cd /etc/init.d      //Enter directory
vi elasticsearch    //Create es system startup service file

(c) Write startup script

#!/bin/bash
#chkconfig: 345 63 37
#description: elasticsearch
#processname: elasticsearch-5.4.0

export ES_HOME=/opt/elasticsearch     #This directory is the directory of your Es folder

case $1 in
        start)
                su elk<<!        #Es this is the account that starts es. If your account is not this, remember to adjust it
                cd $ES_HOME
                ./bin/elasticsearch -d -p pid
                exit
!
                echo "elasticsearch is started"
                ;;
        stop)
                pid=`cat $ES_HOME/pid`
                kill -9 $pid
                echo "elasticsearch is stopped"
                ;;
        restart)
                pid=`cat $ES_HOME/pid`
                kill -9 $pid
                echo "elasticsearch is stopped"
                sleep 1
                su elk<<!     #Es this is the account that starts es. If you don't have this account, remember to adjust it
                cd $ES_HOME
                ./bin/elasticsearch -d -p pid
                exit
!
                echo "elasticsearch is started"
        ;;
    *)
        echo "start|stop|restart"
        ;;
esac
exit 0

(d) Modify file permissions

chmod 777 elasticsearch

(e) Add and remove services and set startup mode

chkconfig --add elasticsearch    //Add system service
chkconfig --del elasticsearch    //Delete system service

You only need to add a system service. If you don't want to use the service to delete it, the two are not executed in sequence.

(f) Shut down and start services

service elasticsearch start     [[start]
service elasticsearch stop      [Stop]
service elasticsearch restart     [Restart]

(g) Set whether the service is started

chkconfig elasticsearch on      [[open]
chkconfig elasticsearch off       [[Close]

Problems encountered

ERROR: [2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Solutions can refer to: Summary of problems and solutions during ELK construction

3. Install kibana

Brief introduction

Kibana is an open source analysis and visualization platform designed for use with Elasticsearch. You can use kibana to search and view the data stored in Elasticsearch. Kibana interacts with Elasticsearch through various charts, tables, maps, etc. to intuitively display data, so as to achieve the purpose of advanced data analysis and visualization.

To put it bluntly, it is a visualization tool, which is similar to Navicat to some extent.

For more information, please refer to: https://www.elastic.co/cn/kibana

Step 1: upload and unzip

Still upload the compressed package to the / opt path, and then decompress it:

//decompression 
tar -zxvf kibana-7.8.1-linux-x86_64.tar.gz

//rename
mv kibana-7.8.1 kibana

Step 2: modify the configuration file

Modify kibana. In the config directory of the configuration YML file, configure elasticsearch address and kibana address information:

server.host: "192.168.92.50"  # kibana server address
elasticsearch.url: "http://192.168.92.50:9200 "# es address

If you want to convert kibana interface into Chinese, you need to add it in the yml file (the last line itself, only comments):

i18n.locale: "zh-CN"

Step 3: start

Start kibana, open / kibana/bin, and run the command:

./kibana   //Display start

nohup /opt/kibana/bin/kibana &   //Background start

Open in browser after startup: http://127.0.0.1:5601 , if it is not installed locally, just change the ip address.

Step 4: view the process

You can't view it through ps -e|grep kibana command. You need to use the process number or search name as follows:

# Query by process number
netstat -tunlp|grep 5601

# Search by name
ps  -aux | grep  kibana

4. Install logstash

Brief introduction

Logstash is an open source data collection engine with real-time pipeline. It can dynamically unify the data from different sources and classify the data to different destinations. It is also a tool for managing events and logs. You can use it to collect logs, analyze them, and store them for later use.

Logstash is usually used with Kibana and Elasticsearch, but logstash does not have to be installed in ELK.

Step 1: upload and unzip

Still upload the compressed package to the / opt path, and then decompress it:

//decompression 
tar -zxvf logstash-7.8.1.tar.gz

//rename
mv logstash-7.8.1 logstash

Step 2: modify the configuration

The key to logstash is configuration. Open the / opt/logstash/config / directory, and then create a configuration file according to your needs.

For example, my requirement is to use logstash to read one json file, and then send its contents to Elasticsearch, then I will create a configuration file: logstash Conf (optional name)

The contents of the document are as follows:

#Read json file
input {
  file {
    #Set the json file path. Multiple file paths can be set as an array [], which can be used for fuzzy matching*
    #Specify a single file
    path => "/opt/suricataFile/eve.json"
    #Specify array file
    #path => ["/data/es/jsonstash-5.6.1/files/test-1.json","/data/es/jsonstash-5.6.1/files/test-2.json"]
    #Specify sibling fuzzy matching
    #path => "/data/es/jsonstash-5.6.1/files/test*.json"
    #Specify multilevel directory fuzzy matching
    #path => "/data/es/jsonstash-5.6.1/files/**/test*.json"

    start_position => "beginning"

    #Set code
    codec => json {charset => "UTF-8"}

    #When there are multiple files, you can use type to specify the input and output path
    type => "json_index"

    # Delete the previously sent data and read it from the beginning every time
    sincedb_path => "/dev/null"
  }
}

#2. Filtering and formatting data stage
filter {

    mutate{
        #Delete invalid field
        remove_field => ["@version","message","host","path"]
    }

    #Add a timestamp field to increase @ timestamp time by 8 hours
    ruby { code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)" }

}

#3. Data output to ES stage
output {

    #Log output format, json_lines;rubydebug, etc
    stdout {
        codec => rubydebug
    }

    # Output to es
    if[type] == "json_index"{

        #Unresolved json is not recorded in elasticsearch
        if "_jsonparsefailure" not in [tags] {
            elasticsearch {
                #es address ip port
                hosts => "127.0.0.1:9200"
                #Indexes
                index => "json_index"
                #type
                document_type => "json_index"

                #Overwrite the template without commenting out. Download the general template: https://download.csdn.net/download/alan_liuyue/11241484
                #template_overwrite=>true
                #template=>"/data/es/logstash-5.6.1/template/logstash.json"

            }
        }
    }

}

The main contents of the configuration file are: read the contents of "/ opt/suricataFile/eve.json" file, and then send it to the index JSON of Elasticsearch after filtering_ Index. If there is no index, create the index.

The notes above are very detailed. I won't say more about them.

Step 3: start logstash

The operation command is as follows:

/opt/logstash/bin/logstash -f /opt/logstash/config/logstash.conf

Run the command to specify the configuration file (created in step 2).

Problems encountered

Stuck in: successfully started logstash API endpoint {: Port = > 9600}

As shown in the figure above, it's stuck here. I thought it was successful at first, but when I went to query the index, there was no addition or import. I don't know what the problem is, and I changed the configuration file.

Finally, refer to the following two blogs to solve the problem:

The reason for this problem is that logstash reads Eve When the JSON file is read completely, but it is not sent (there may be a bug, I don't know for the time being). At this time, a sincedb file is generated, and eve.exe is read every time in the future JSON files are read from the end, so they can't be read.

Sincedb needs to be added_ Path = > "/ dev/null". This is the information obtained by the first blog, but it doesn't say where the increase is?

The second blog knows where to increase, so it solves the problem smoothly.

Do you need to write user and password in the configuration file

When solving the first problem, I read many logstash configuration files and found that some blogs mentioned that the user name and password of the host where Elasticsearch is located should be written in the configuration file.

However, after the above problem is finally solved, run it again in the root user (the user and password are not written in the configuration file). It is found that it can still run, indicating that it is not necessary

Keywords: ElasticSearch ELK LogStash kibana

Added by jeff21 on Tue, 08 Feb 2022 23:40:16 +0200