Construction and use of ELK real-time diary platform

1. What is ELK
   ELK is the acronym of Elasticsearch, logstash and Kibana (but the later filebeat (one of beats) can be used to replace the data collection function of logstash, which is lightweight). It is also known as Elastic Stack on the market.

   Elasticsearch: it is the core distributed search and analysis engine of ElasticStack. It is a near real-time search platform framework based on Lucene, distributed and interactive through Restful mode. Elasticsearch provides near real-time search and analysis for all types of data. Whether you are structured or unstructured text, digital data or geospatial data, elasticsearch can effectively store and index it in a way that supports fast search.
   Logstash: it is a free and open server-side data processing pipeline, which can collect data from multiple sources, convert data, and then send data to your favorite "repository". Logstash can dynamically collect, convert and transmit data without being affected by format or complexity. Grok is used to derive structure from unstructured data, decode geographical coordinates from IP address, anonymize or exclude sensitive fields, and simplify the overall processing process.

   Filebeat: a lightweight delivery tool for forwarding and centralizing log data. Filebeat monitors the log files or locations you specify, collects log events, and forwards them to Elasticsearch or Logstash for indexing. Filebeat works as follows: when you start filebeat, it will start one or more inputs that will be found in the location specified for log data. For each log found by filebeat, filebeat starts the collector. Each collector reads a single log to get new content and sends the new log data to libbeat, which aggregates events and sends the aggregated data to the output configured for filebeat.

   Kibana: an open source analysis and visualization platform for ElasticSearch, which is used to search and view the data interactively stored in the ElasticSearch index. With kibana, advanced data analysis and display can be carried out through various charts. And it can provide log analysis friendly Web interface for Logstash and ElasticSearch, which can summarize, analyze and search important data logs. It can also make massive data easier to understand. It is easy to operate, and the browser based user interface can quickly create a dashboard to display the ElasticSearch query dynamics in real time.

2. Why use ELK
   Logs mainly include system logs, application logs and security logs. System operation and maintenance personnel and developers can understand the software and hardware information of the server through the log, check the errors in the configuration process and the causes of the errors. Regular analysis of logs can understand the load, performance and security of the server, so as to take timely measures to correct errors. Often, the logs of a single machine can be basically analyzed by using grep, awk and other tools, but when the logs are scattered and stored on different devices. If you manage dozens or hundreds of servers, you are still using the traditional method of logging in to each machine in turn. Does this feel cumbersome and inefficient. It is imperative that we use centralized log management, such as the open source syslog, to collect and summarize the logs on all servers. After centralized log management, log statistics and retrieval become a more troublesome thing. Generally, we can use grep, awk, wc and other Linux commands to achieve retrieval and statistics, but we still have a little difficulty in using this method for higher requirements such as query, sorting and statistics and a large number of machines. Generally, a large-scale system is a distributed deployment architecture. Different service modules are deployed on different servers. When a problem occurs, it is necessary to locate the specific server and service module according to the key information exposed by the problem, and build a centralized log system, which can improve the efficiency of locating the problem.

3. How to use ELK

3.1 installation on docker

3.1. 1 deploy ELK
Learning documents: https://elk-docker.readthedocs.io/#usage

(1) Prerequisites
sysctl vm.max_map_count needs to be 262144

vim /etc/sysctl.conf
vm.max_map_count = 262144

Add VM max_ map_ count = 262144

sysctl -p 

(2) Pull the image in docker

docker pull sebp/elk

(3) Start elk

docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

(3) Startup elk with configuration file (I seem to have read the wrong document)
Source document:
#Open three ports to transfer information, mount all files under the three paths of the host, rename the container elk, and the image sebp/elk
Learning documents: https://blog.csdn.net/qq_40673345/article/details/103567305

docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -v /opt/elk-data/conf:/opt/conf  -v /opt/elk-data/logs:/opt/logs  -v /opt/elk-data/elasticsearch-data:/var/lib/elasticsearch  -it -d --name elk sebp/elk

5601 (Kibana network interface).
9200 (Elasticsearch JSON interface).
5044 (Logstash Beats interface, which receives logs from Beats, such as Filebeat).
Close ^ C
Start docker start elk
Close docker stop elk

3.1. 2 use ELK https://blog.csdn.net/qq_34988304/article/details/100058049

(1) Introduce dependency

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>5.2</version>
</dependency>

(2) Configure logback-spring.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" scanPeriod="60 seconds">
    <!-- Define parameters -->
    <property name="log.pattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n" />
    <!-- Console print settings -->
    <appender name="consoleAppender" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>${log.pattern}</pattern>
        </encoder>
    </appender>

    <!-- logstash set up -->
    <appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <param name="Encoding" value="UTF-8"/>
        <!-- logstash The server ip -->
        <remoteHost>192.168.160.4</remoteHost>
        <!-- logstash tcp port-->
        <port>5044</port>
        <!-- <filter class="com.program.interceptor.ELKFilter"/>-->
        <!-- encoder is required -->
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder" >
            <customFields>{"appname":"courseService"}</customFields> 
        </encoder>
    </appender>

    <root level="info">
        <appender-ref ref="consoleAppender" />
        <appender-ref ref="logstash"/>
    </root>
</configuration>

On linux

deploy
Learning documents: https://www.cnblogs.com/zsql/p/13164414.html

Use the same as above

logstash auto reload configuration
Learning documents: https://blog.csdn.net/weixin_34032792/article/details/86259176