1. Introduction to SaltStack

1.1 automatic operation and maintenance tools

Automatic operation and maintenance is to hand over periodic, repetitive and regular work to tools to do, and turn the past manual execution into automatic operation. Automation is the sublimation of IT operation and maintenance. IT operation and maintenance automation is not only a maintenance process, but also a management improvement process. IT is not only the highest level of IT operation and maintenance, but also the development trend in the future

Common automatic operation and maintenance tools

name	advantage	shortcoming
SaltStack	The status file can be a simple YAML configuration module or a complex Python/PyDSL script Clients can install agents based on SSH or on managed nodes The Web interface can see the running work, minion status, event log and commands that can be executed on the client Strong expansion ability	The Web interface is like poison. The competitive products are unstable and relatively imperfect Lack of ability to generate in-depth reports
Puppet	Modules are written in Ruby or a subset of Ruby The push command can trigger the change Web interface generation and processing report, resource list and real-time node management The agent running end makes detailed and in-depth reports and configures nodes	Compared with other tools, it is more complex. You need to learn Puppet's DSL or Ruby The installation process lacks error checksum and generates error report
Ansible	Modules can be developed in any language The standby node does not need to install agent software There are Web management interface, configurable users, groups, resource list and Playbook execution Installation and operation are extremely simple	The standby management node is Windows, which needs to be strengthened The Web management interface is part of the built-in Ansible List of resources to be imported Low execution efficiency

1.2 SaltStack service architecture

In the saltstack architecture, the server side is called Master and the client side is called Minion.

Both the Master and minion operate in the mode of daemon, and always listen to ret defined in the configuration file_ Port (accept minion request) and publish_ Port.

When Minion is running, it will automatically connect to the Master address RET defined in the configuration file_ Port port for connection authentication.

In addition to the traditional C/S architecture, saltstack also has an architecture called masterless, which does not need to install a master server separately. It only needs to install the Minion end on each machine, and then adopts the mode that the machine is only responsible for the configuration management mechanism service of the machine.

2. Four functions and four operation modes of saltstack

saltstack has four functions:

• Remote execution
• Configuration management / status management
• cloud management
• event driven

saltstack can realize batch management through remote execution, and achieve the purpose of some functions by describing the state.

saltstack has four operation modes:

• local run locally
• Master/Minion traditional mode
• Syndic distributed
• Salt ssh

3. saltstack component introduction

assembly	function
Salt Master	Used to send commands and configurations to the Salt minion running on the managed system
Salt Minions	Receive commands and configurations from the Salt master
Execution Modules	A temporary command executed from the command line for one or more managed systems
Formulas (States)	Declarative or imperative representation of system configuration
Grains	Grains is static information about the underlying managed system, including operating system, memory and many other system properties.
Pillar	User defined variables. These security variables are defined and stored in the Salt Master and then "assigned" to one or more minions using the target. Pillar data stores values such as ports, file paths, configuration parameters, and passwords.
Top File	Match Formulas (States) and Salt Pillar data with Salt minions.
Runners	The module executed on the Salt master is used to perform support tasks. Salt runners report job status, connection status, read data from external API s, query connected Salt minions, etc.
Returners	Send the data returned by Salt minions to another system, such as a database. Salt Returners can run on Salt minion or Salt master.
Reactor	Triggers a response when an event occurs in the SaltStack environment.
Salt Cloud / Salt Virt	Provide the system on the cloud provider / hypervisor and put it under management immediately.
Salt SSH	Run the Salt command through SSH on a system without Salt minion.

4. SaltStack deployment

Environmental description

host name	IP	service	System version
JLin	192.168.23.116	salt-master salt-minion	CentOS 8 Stream
JiaL	192.168.23.117	salt-minion	CentOS 8 Stream

Official yum source address

https://repo.saltproject.io

4.1 installing Saltstack on the control node

Configure yum source

[root@JLin ~]# rpm --import https:// repo.saltproject.io/py3/redhat/8/x86_64/latest/SALTSTACK-GPG-KEY.pub
[root@JLin ~]# curl -fsSL https:// repo.saltproject.io/py3/redhat/8/x86_64/latest.repo | sudo tee /etc/yum.repos.d/salt.repo
[salt-latest-repo]
name=Salt repo for RHEL/CentOS 8 PY3
baseurl=https:// repo.saltproject.io/py3/redhat/8/x86_64/latest
skip_if_unavailable=True
failovermethod=priority
enabled=1
enabled_metadata=1
gpgcheck=1
gpgkey=https:// repo.saltproject.io/py3/redhat/8/x86_64/latest/SALTSTACK-GPG-KEY.pub
[root@JLin ~]# yum clean all

Install SaltStack master

[root@JLin ~]# yum -y install salt-master salt-minion
[root@JLin ~]# rpm -qa | grep salt
salt-minion-3004-1.el8.noarch
salt-3004-1.el8.noarch
salt-master-3004-1.el8.noarch

Modify profile

[root@JLin ~]# vim /etc/salt/minion
# Set the location of the salt master server. If the master server cannot be
# resolved, then the minion will fail to start.
#master: salt
master: 192.168.23.180

Start service

[root@JLin ~]# systemctl enable --now salt-master.service
[root@JLin ~]# systemctl enable --now salt-minion.service 
[root@JLin ~]# ss -anltup | grep salt
tcp   LISTEN 0      128           0.0.0.0:4505       0.0.0.0:*    users:(("salt-master",pid=48716,fd=22))                  
tcp   LISTEN 0      128           0.0.0.0:4506       0.0.0.0:*    users:(("salt-master",pid=48722,fd=35)) 

4.2 install salt minion client on the controlled machine

Configure yum source

[root@JiaL ~]# rpm --import https:// repo.saltproject.io/py3/redhat/8/x86_64/latest/SALTSTACK-GPG-KEY.pub
[root@JiaL ~]# curl -fsSL https:// repo.saltproject.io/py3/redhat/8/x86_64/latest.repo | sudo tee /etc/yum.repos.d/salt.repo
[salt-latest-repo]
name=Salt repo for RHEL/CentOS 8 PY3
baseurl=https:// repo.saltproject.io/py3/redhat/8/x86_64/latest
skip_if_unavailable=True
failovermethod=priority
enabled=1
enabled_metadata=1
gpgcheck=1
gpgkey=https:// repo.saltproject.io/py3/redhat/8/x86_64/latest/SALTSTACK-GPG-KEY.pub
[root@JiaL ~]# yum clean all

Install salt minion

[root@JiaL ~]# yum -y install salt-minion
[root@JiaL ~]# rpm -qa | grep salt
salt-3004-1.el8.noarch
salt-minion-3004-1.el8.noarch

Modify the configuration file of the controlled end and set the master as the IP of the control node

[root@JiaL ~]# vim /etc/salt/minion
# Set the location of the salt master server. If the master server cannot be
# resolved, then the minion will fail to start.
#master: salt
master: 192.168.23.180

Start service

[root@JiaL ~]# systemctl enable --now salt-minion.service 

4.3 saltstack configuration file

The configuration file of saltstack is in / etc/salt directory

saltstack configuration file description:

configuration file	explain
/etc/salt/master	Master (control) configuration file
/etc/salt/minion	Controlled end profile

The configuration file / etc/salt/master works well with the default configuration, so there is no need to modify this configuration file.
Configuration file / etc/salt/minion common configuration parameters

• Master: sets the IP address of the master
• id: set the unique identifier of the local machine of the controlled end, which can be ip, host name or a meaningful word

In the process of daily use, it is often necessary to adjust or modify the Master configuration file. Most configurations of saltstack have specified default values, which can be modified according to their actual needs. The following parameters are important

• max_open_files: the number of minions can be adjusted appropriately according to the Master
• timeout: it can be adjusted according to the network conditions of Master and Minion
• auto_accept and autosign_file: automatic visa can be set when Minion is deployed on a large scale
• master_ Top and all parameters starting with external: these parameters are related to the integration of SaltStack with external systems

5. saltstack authentication mechanism

The saltstack master relies on the openssl certificate to authenticate and communicate with the host of the controlled end. After the controlled end is started, it will send a public key certificate file to the master and use the salt key command to manage the certificate at the master.

Authentication process of salt minion and salt master:

• When minion is started for the first time, it will automatically generate a pair of keys under / etc/salt/pki/minion /, and then send the public key to the master
• After receiving the public key of minion, the master accepts the public key through the salt key command. At this time, the / etc/salt/pki/master/minions directory of the master will store the public key named minion id, and then the master can send control instructions to minion

Salt key common options

-L      // List all public key information
-a minion    // Accept the key of the specified minion waiting for authentication
-A      // Accept all minion key s waiting for authentication
-r minion    // Reject the specified minion key waiting for authentication
-R      // Reject all minion key s waiting for authentication
-f minion   // Displays the fingerprint information of the specified key
-F      // Display fingerprint information of all key s
-d minion   // Delete the key of the specified minion
-D      // Delete all minion key s
-y      // Automatic answer yes

View current certificate status

[root@JLin ~]# salt-key -L
Accepted Keys:
JLin
Denied Keys:
Unaccepted Keys:
JiaL
Rejected Keys:

Accept the new certificate for the specified minion

[root@JLin ~]# salt-key -ya JiaL
The following keys are going to be accepted:
Unaccepted Keys:
JiaL
Key for minion JiaL accepted.
[root@JLin ~]# salt-key -L
Accepted Keys:
JLin
JiaL
Denied Keys:
Unaccepted Keys:
Rejected Keys:

6. saltstack remote execution

Tests whether the specified managed host is alive

[root@JLin ~]# salt "JiaL" test.ping
JiaL:
    True
[root@JLin ~]# salt "JiaL" cmd.run hostname
JiaL:
    JiaL
[root@JLin ~]# salt "*" test.ping
JLin:
    True
JiaL:
    True

7. salt command use

// Syntax: salt [options] '< target >' < function > [arguments]

// Common options
    --version       // View the version number of saltstack
    --versions-report   // View the version number of saltstack and dependent packages
    -h      // View help information
    -c CONFIG_DIR   // Specify the configuration file directory (default is / etc/salt /)
    -t TIMEOUT      // Specify timeout (default is 5s)
    --async     // Asynchronous execution
    -v      // verbose mode, which displays the execution process in detail
    --username=USERNAME     // Specify external authentication user name
    --password=PASSWORD     // Specify external authentication password
    --log-file=LOG_FILE     // Specify logging file
    
// Common target parameters
    -E      // Regular matching
    -L      // List matching 
    -S      // CIDR matching network segment
    -G      // grains matching
    --grain-pcre    // grains plus regular matching
    -N      // Group matching
    -R      // Range matching
    -C      // Comprehensive matching (specify multiple matches)
    -I      // pillar value match

Examples

// Regular matching
[root@JLin ~]# salt -E 'J*' test.ping
JLin:
    True
JiaL:
    True
// List matching

Modify id

// Control node
[root@JLin ~]# salt-key -yd JiaL
The following keys are going to be deleted:
Accepted Keys:
JiaL
Key for minion JiaL deleted.
[root@JLin ~]# salt-key -L
Accepted Keys:
JLin
Denied Keys:
Unaccepted Keys:
Rejected Keys:

// Managed host
[root@JiaL ~]# cd /etc/salt/pki/minion/
[root@JiaL minion]# ls
minion.pem  minion.pub  minion_master.pub
[root@JiaL minion]# rm -rf *
[root@JiaL minion]# vim /etc/salt/minion
# Explicitly declare the id for this minion to use, if left commented the id
# will be the hostname as returned by the python call: socket.getfqdn()
# Since salt uses detached ids it is possible to run multiple minions on the
# same machine but with different ids, this can be useful for salt compute
# clusters.
#id:
id: 192.168.23.181
[root@JiaL minion]# systemctl restart salt-minion.service
[root@JiaL minion]# ls
minion.pem  minion.pub

// Control node
[root@JLin ~]# salt-key -L
Accepted Keys:
JLin
Denied Keys:
Unaccepted Keys:
192.168.23.181
Rejected Keys:
[root@JLin ~]# salt-key -ya 192.168.23.181
The following keys are going to be accepted:
Unaccepted Keys:
192.168.23.181
Key for minion 192.168.23.181 accepted.
[root@JLin ~]# salt-key -L
Accepted Keys:
192.168.23.181
JLin
Denied Keys:
Unaccepted Keys:
Rejected Keys:
[root@JLin ~]# salt '*' test.ping
JLin:
    True
192.168.23.181:
    True
[root@JLin ~]# salt -G 'os:centos stream' test.ping
192.168.23.181:
    True
JLin:
    True