Microservice series: a guide to building distributed log ELK

From the service gateway: Microservices series: introduction to Spring Cloud Gateway I've always learned about distributed transactions: Microservices series: distributed transaction Spring Cloud Alibaba's Seata practical chapter Today, we are finally going to end our micro service series. Not surprisingly, distributed logging will be the last article in the current microservice series, which may be supplemented in the future.

Without much to say, I will begin to learn the last article of the current microservice series.

1, Basic introduction

1. What is distributed logging

I believe that in your daily work, you will often encounter many exceptions and error messages in the production environment. You need to check the log information to check the errors. Moreover, this project is a distributed application, and the logs are scattered on different storage devices. If you manage dozens or hundreds of servers, you can also use the traditional method of logging in to each machine in turn to check the log. Does this feel cumbersome and inefficient. Therefore, we use centralized log management. Distributed log is to collect, track and process large-scale log data.

Generally, we need to conduct log analysis scenarios: you can get the information you want by directly grep and awk in the log file. However, in large-scale scenarios, this method is inefficient and faces problems, including how to archive if the log volume is too large, what to do if the text search is too slow, and how to query in multiple dimensions. Centralized log management is required, and the logs on all servers are collected and summarized. The common solution is to establish a centralized log collection system to collect, manage and access the logs on all nodes.

2. ELK distributed log

In fact, ELK is the abbreviation of three software, which are composed of Elasticsearch, Logstash and Kibana.

Elasticsearch is an open source distributed search engine based on java. Its characteristics include: distributed, zero configuration, automatic discovery, automatic index fragmentation, index copy mechanism, restful style interface, multiple data sources, automatic search load, etc.

Kibana is an open source and free tool based on nodejs. Kibana can provide log analysis friendly Web interface for Logstash and ElasticSearch, and can summarize, analyze and search important data logs.

Based on java, Logstash is an open source tool for collecting, analyzing and storing logs.

Here is how ELK works:

Suppose that our project is deployed on three servers. Generally, Elasticsearch should be installed on all three servers. Kibana should be installed on the server as the master node, and Logstash should be installed on one of the node servers.

Next, let's build it.

2, Build ELK

During the construction process, you must pay attention to the correspondence between versions. If the versions do not correspond, various startup errors will appear.

Version Description:

  • CentOS 7.9
  • JDK 1.8
  • Elasticsearch-8.0.0
  • Logstash-8.0.0
  • Kibana-8.0.0

1. Elasticsearch


Elasticsearch is a Lucene based search server. It provides a distributed multi-user full-text search engine based on RESTful web interface. Elasticsearch is developed in Java and released as an open source under the Apache license terms. It is a popular enterprise search engine.

1.2 installation configuration

First, go to the official website to download: Download Elasticsearch | Elastic , select the Linux version

PS: the download speed of the official website is still very fast

This time I use CentOS 7 in the virtual machine. First copy the downloaded tar package to / usr/local in the virtual machine,

  • decompression
tar -zxvf elasticsearch-8.0.0-linux-x86_64.tar.gz
  • Modify configuration
cd /usr/local/elasticsearch-8.0.0/config/
vim elasticsearch.yml
cluster.name: my-application
node.name: node-1
path.data: /home/esuser/data	# Data file storage directory
path.logs: /home/esuser/logs	# Log file storage directory
network.host:			# Open to all IP
http.port: 9200					# Port number
discovery.seed_hosts: [""]
cluster.initial_master_nodes: ["node-1"]

Note: path Data and path The logs directory needs to be created manually mkdir

Note: these configurations are annotated by default. After modification, open the annotation

In addition, because we are only testing and learning this time, we can turn off the security related configuration from true to false

# Enable security features
xpack.security.enabled: false
xpack.security.enrollment.enabled: false
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
  enabled: false
  keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
  enabled: false

Save and exit editing after modification

  • Create esuser user

Because ElasticSearch does not support direct operation by Root users, we need to create an esuser user

# Create user
useradd esuser
# Set password
passwd esuser
# Give user permission
chown -R esuser:esuser /usr/local/elasticsearch-8.0.0

1.3 startup

Switch user to esuser user for operation

su esuser
cd /usr/local/elasticsearch-8.0.0/bin
./elasticsearch -d & tail -f /home/esuser/logs/my-application.log 	# Start in the background and view the startup log

If there is no error at startup, the host browser access address is: is the static ip of the virtual machine. The successful access is shown in the figure below (pay attention to the firewall)

2. Logstash

2.1 introduction

Logstash is an open source server-side data processing pipeline, which can collect data from multiple sources at the same time, convert data, and then send data to the repository. Of course, our repository in this article is ElasticSearch.

2.2 installation configuration

Download address of official website: Download Logstash Free | Get Started Now | Elastic

Copy the downloaded tar package to / usr/local in the virtual machine,

  • decompression
tar -zxvf logstash-8.0.0-linux-x86_64.tar.gz
  • New profile
cd /usr/local/logstash-8.0.0/bin
touch logstash-elasticsearch.conf
vim logstash-elasticsearch.conf
input {
	stdin {}
output {
	elasticsearch {
		hosts => ''
	stdout {
		codec => rubydebug

Note the address of elasticsearch inside

2.3 startup

At this time, it is under the root user

cd /usr/local/logstash-8.0.0/bin
nohup ./logstash -f logstash-elasticsearch.conf &		# Background start

Note: execute the background startup command to view nohup It is found that there is an error when logging out, but it seems that it has been started successfully, which will not affect the use. If nohup is not used, there will be no error.

Use jps to view all current java processes

Found that the startup was successful.

3. Kibana

3.1 introduction

Kibana is an open source data analysis and visualization platform. It is one of the members of Elastic Stack and is designed to cooperate with elastic search. You can use kibana to search, view and interact with the data in Elasticsearch index. You can easily use charts, tables and maps to conduct diversified analysis and presentation of data.

3.2 installation configuration

Download address of official website: Download Kibana Free | Get Started Now | Elastic

Copy the downloaded tar package to / usr/local in the virtual machine,

  • decompression
tar -zxvf kibana-8.0.0-linux-x86_64.tar.gz
  • Modify configuration
cd /usr/local/kibana-8.0.0/config/
vim kibana.yml
server.port: 5601			# Port number
server.host: ""		# Open access to all ip addresses
server.name: "my-kibana"
elasticsearch.hosts: [""] 		#  es address
i18n.locale: "zh-CN"	# Sinicization

Note: these configurations are annotated by default. After modification, open the annotation

  • kibana can't be started by the root user, so we use the esuser user created earlier and authorize it first
chown -R esuser:esuser /usr/local/kibana-8.0.0

3.3 startup

Switch to esuser user

su esuser

Background start

/usr/local/kibana-8.0.0/bin &

Successful startup, browser access address:

The following page appears, indicating that the startup is successful

We click the applicable sample data and click add data

Click Discover

You can see the sample data

So far, the software we need for distributed logging has been installed, configured and started successfully

As for how to use it, I feel I need an article to write.

Keywords: Distribution Microservices ELK

Added by knelson on Wed, 23 Feb 2022 03:24:48 +0200