Elasticsearch 7.1.1 cluster construction

1 prepare the installation environment

1.1 installing JDK

elasticsearch 7.1.1 configuring java8, java11

1.2 change system resource configuration

Modify / etc / sysctl Conf file, add VM. Conf at the end of the file max_ map_ count=262144

Note: after modification, execute sysctl -p, and load system parameters from the specified file. If not specified, start / etc / sysctl Loading in conf

View the result sysctl - a|grep VM max_ map_ count

Modify / etc / security / limits Conf file

# Add the following permanent system tuning at the end of the file, modify the file descriptor size (65536) and the maximum number of processes
# *Represents the user name of all services. Users can also be set, such as esuser
* soft nofile 65536
* hard nofile 65536
* soft nproc 4096
* hard nproc 4096

2. Install Elasticsearch cluster

2.1 preparing cluster configuration

The three machines are used to save data and can be selected as the master node

Machine ip	master node	data node	edition
172.16.153.3	yes	yes	7.1.1
172.16.193.220	yes	yes	7.1.1
172.16.193.239	yes	yes	7.1.1

2.1.1 create esuser user

# Add user group
groupadd esuser

# Add user
useradd -m -g esuser esuser

# Configure Ciphers
passwd esuser

# Add sudo permission at the end
esuser ALL=(ALL)  NOPASSWD:ALL

2.2 download & install

# download
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.1.1-linux-x86_64.tar.gz

# decompression
tar xvf elasticsearch-7.1.1-linux-x86_64.tar.gz

# Soft chain
ln -s elasticsearch-7.1.1 elasticsearch

2.3 configuration

2.3.1 configuration description

parameter	explain
cluster.name	Cluster name. The same name is a cluster
node.name	Node name. Each node name is unique in cluster mode
node.master	Whether the current node can be elected as a master node: true, No: false
node.data	Whether the current node is used to store data. Yes: true, No: false
path.data	Location of index data
path.logs	Location of log files
bootstrap.memory_lock	Physical memory needs to be locked. Yes: true, No: false
bootstrap.system_call_filter	SecComp detection, yes: true, No: false
network.host	Listening address, used to access the es
network.publish_host	It can be set as intranet ip for communication between machines in the cluster
http.port	The http port provided by es is 9200 by default
discovery.seed_hosts	es7. For the configuration added after X, write the device address of the candidate master node. After the service is started, it can be selected as the master node
cluster.initial_master_nodes	es7.x. This configuration is required to elect a master when initializing a new cluster
http.cors.enabled	Whether cross domain is supported is: true. This configuration is required when using the head plug-in
http.cors.allow-origin	"*" indicates that all domain names are supported

2.3.2 detailed configuration

172.16.153.3 configuration

# 7.1.1 configuration

# Add the following:
cluster.name: my-application

node.name: node-1
node.master: true
node.data: true

path.data: /data/es/9200/data1,/data/es/9200/data2,/data/es/9200/data3
path.logs: /data/es/9200/logs

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

network.host: 173.16.153.3
# Sometimes this configuration is not required, but I need it here
# network.publish_host: 173.26.153.48
transport.tcp.port: 9300
http.port: 9200
# discovery.seed_hosts: ["173.26.153.48","173.16.153.252","173.16.153.161"]
discovery.seed_hosts: ["173.16.153.3:9300","173.16.193.220:9300","173.16.193.239:9300"]
cluster.initial_master_nodes: ["173.16.153.3","173.16.193.220","173.16.193.239"]
gateway.recover_after_nodes: 2
cluster.routing.allocation.disk.threshold_enabled: false


http.cors.enabled: true
http.cors.allow-origin: "*"

172.16.193.220 configuration

# 7.1.1 configuration

# Add the following:
cluster.name: my-application

node.name: node-2
node.master: true
node.data: true

path.data: /data/es/9200/data1,/data/es/9200/data2,/data/es/9200/data3
path.logs: /data/es/9200/logs

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

network.host: 173.16.193.220
# Sometimes this configuration is not required, but I need it here
# network.publish_host: 173.26.153.48
transport.tcp.port: 9300
http.port: 9200
# discovery.seed_hosts: ["173.26.153.48","173.16.153.252","173.16.153.161"]
discovery.seed_hosts: ["173.16.153.3:9300","173.16.193.220:9300","173.16.193.239:9300"]
cluster.initial_master_nodes: ["173.16.153.3","173.16.193.220","173.16.193.239"]
gateway.recover_after_nodes: 2
cluster.routing.allocation.disk.threshold_enabled: false


http.cors.enabled: true
http.cors.allow-origin: "*"

172.16.193.239 configuration

# 7.1.1 configuration

# Add the following:
cluster.name: my-application

node.name: node-3
node.master: true
node.data: true

path.data: /data/es/9200/data1,/data/es/9200/data2,/data/es/9200/data3
path.logs: /data/es/9200/logs

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

network.host: 173.16.193.239
# Sometimes this configuration is not required, but I need it here
# network.publish_host: 173.26.153.48
transport.tcp.port: 9300
http.port: 9200
# discovery.seed_hosts: ["173.26.153.48","173.16.153.252","173.16.153.161"]
discovery.seed_hosts: ["173.16.153.3:9300","173.16.193.220:9300","173.16.193.239:9300"]
cluster.initial_master_nodes: ["173.16.153.3","173.16.193.220","173.16.193.239"]
gateway.recover_after_nodes: 2
cluster.routing.allocation.disk.threshold_enabled: false


http.cors.enabled: true
http.cors.allow-origin: "*"

2.3.3 modify the JVM option

-Xms10g
-Xmx10g

2.3.4 add es installation directory permission

chown -R esuser:esuser /opt/es

2.3.5 start the cluster

./elasticsearch

# Background operation
./elasticsearch -d

The successful startup interface is as follows:

2.3.6 setting cluster authentication password

2.3.6.1 generate certificate

172.16.153.3 on the machine:

bin/elasticsearch-certutil cert -out config/elastic-certificates.p12 -pass ""

Note: the password needs to be set separately. Here is the cluster security authentication. It is recommended not to set the password. The certificate generated after success is in the config directory of es by default p12； Copy a copy to the config of other nodes (default directory)

2.3.6.2 in elasticsearch YML configuration add configuration

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

2.3.6.3 restart the cluster and change the password

bin/elasticsearch-setup-passwords interactive

elastic account: it has the super user role and is a built-in super user.
Kibana account: own kibana_system role. Kibana is used to connect and communicate with elasticsearch. The kibana server submits a request as the user to access the cluster monitoring API and Kibana index. Cannot access index.
logstash_system account: own logstash_system role. The user Logstash is used when storing monitoring information in Elasticsearch.
beats_system account: owned beats_system role. User Beats is used when storing monitoring information in Elasticsearch.

3 install Kibana

# kibana.yml configuration

server.port: 5601
server.host: "173.26.153.48"
elasticsearch.hosts: ["http://173.16.153.3:9200","http://173.16.193.220:9200","http://173.16.193.239:9200"]
elasticsearch.username: "elastic"
elasticsearch.password: "123456"

Attachment: complete configuration