1. return of saltstack component
The return component can be understood as the SaltStack system stores or returns the data returned by Minion to other programs. It supports a variety of storage methods, such as MySQL, MongoDB, Redis, Memcache, etc. through return, we can record each operation of SaltStack and provide a data source for future log audit. At present, 30 return data storage and interfaces are officially supported. We can easily configure and use it. Of course, it also supports self-defined returns. Custom returns need to be written in python. After selecting and configuring the return to use, just specify return after the salt command.
//example
[root@master ~]# salt 'node1' sys.list_returners
node1:
- carbon
- couchdb
- etcd
- highstate
- local
- local_cache
- mattermost
- multi_returner
- pushover
- rawfile_json
- slack
- slack_webhook
- smtp
- splunk
- sqlite3
- syslog
- telegram
1.1 return process
Return is to trigger the task on the Master side, and then Minion accepts the processing task, directly establishes a connection with the return storage server, and then saves the data return to the storage server. It must be noted that the Minion side operates the storage server in this process, so it is necessary to ensure that the configuration and dependency package of the Minion side are correct, which means that we must install the specified return mode dependency package on each Minion. If Mysql is used as the return storage mode, we will install Python Mysql module on each Minion.
1.2 use mysql as the return storage method
Environmental description:
centos8
| host name | IP address | Deployment function | Realize function |
|---|---|---|---|
| master | 192.168.143.101 | salt-master mariadb python3-PyMySQL | Execution effect of master side and control side |
| node1 | 192.168.143.102 | salt-minion mariadb python3-PyMySQL | Execute command at minion end |
| node2 | 192.168.143.103 | mariadb mariadb-server | return database storage |
Install MySQL Python module on all minion s
[root@master salt]# salt-key -L
Accepted Keys:
node1
Denied Keys:
Unaccepted Keys:
master
node2
node3
node4
node5
Rejected Keys:
[root@master salt]# salt '*' test.ping
node1:
True
[root@node1 ~]# dnf list all|grep -i 'mysql' |grep python
python2-PyMySQL.noarch 0.8.0-10.module_el8.4.0+642+1dc4fb01 appstream
python3-PyMySQL.noarch 0.10.1-2.module_el8.4.0+666+456f5f48 appstream
python38-PyMySQL.noarch 0.10.1-1.module_el8.4.0+677+b84873a2 appstream
python39-PyMySQL.noarch 0.10.1-2.module_el8.4.0+680+7b309a77 appstream
[root@node1 ~]# dnf -y install python3-PyMySQL
Deploy a mysql server as a storage server. Here, deploy it directly on the host 192.168.143.103
[root@node2 ~]# dnf -y install mariadb-server mariadb
[root@node2 ~]# systemctl enable --now mariadb
Created symlink /etc/systemd/system/mysql.service → /usr/lib/systemd/system/mariadb.service.
Created symlink /etc/systemd/system/mysqld.service → /usr/lib/systemd/system/mariadb.service.
Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service → /usr/lib/systemd/system/mariadb.service.
[root@node2 ~]# ss -atnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 80 *:3306 *:*
LISTEN 0 128 *:80 *:*
LISTEN 0 128 [::]:22 [::]:*
[root@node2 ~]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 8
Server version: 10.3.28-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> set password = password("qwer123!");
Query OK, 0 rows affected (0.000 sec)
MariaDB [(none)]> quit
Bye
[root@node2 ~]# mysql -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 9
Server version: 10.3.28-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> quit
Bye
//Create database and table structure on node2
[root@node2 ~]# mysql -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 10
Server version: 10.3.28-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> CREATE DATABASE `salt`
-> DEFAULT CHARACTER SET utf8
-> DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.000 sec)
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| salt |
+--------------------+
4 rows in set (0.003 sec)
MariaDB [(none)]> use salt
Database changed
MariaDB [salt]> DROP TABLE IF EXISTS `jids`;
Query OK, 0 rows affected, 1 warning (0.005 sec)
MariaDB [salt]> CREATE TABLE `jids` (
-> `jid` varchar(255) NOT NULL,
-> `load` mediumtext NOT NULL,
-> UNIQUE KEY `jid` (`jid`)
-> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.006 sec)
MariaDB [salt]> show tables;
+----------------+
| Tables_in_salt |
+----------------+
| jids |
+----------------+
1 row in set (0.000 sec)
MariaDB [salt]> CREATE TABLE `salt_returns` (
-> `fun` varchar(50) NOT NULL,
-> `jid` varchar(255) NOT NULL,
-> `return` mediumtext NOT NULL,
-> `id` varchar(255) NOT NULL,
-> `success` varchar(10) NOT NULL,
-> `full_ret` mediumtext NOT NULL,
-> `alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-> KEY `id` (`id`),
-> KEY `jid` (`jid`),
-> KEY `fun` (`fun`)
-> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.004 sec)
MariaDB [salt]> CREATE TABLE `salt_events` (
-> `id` BIGINT NOT NULL AUTO_INCREMENT,
-> `tag` varchar(255) NOT NULL,
-> `data` mediumtext NOT NULL,
-> `alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-> `master_id` varchar(255) NOT NULL,
-> PRIMARY KEY (`id`),
-> KEY `tag` (`tag`)
-> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.007 sec)
MariaDB [salt]>
//Authorized access
MariaDB [salt]> grant all on salt.* to salt@'%' identified by 'salt';
Query OK, 0 rows affected (0.001 sec)
MariaDB [salt]> flush privileges;
Query OK, 0 rows affected (0.001 sec)
MariaDB [salt]> quit
Configure minion
//Test whether it can be connected on node1 [root@node1 ~]# dnf -y install mariadb [root@node1 ~]# mysql -usalt -psalt -h192.168.143.103 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 11 Server version: 10.3.28-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> quit Bye //Configuration minion file [root@node1 ~]# vim /etc/salt/minion # Default minion returners. Can be a common delimited string or a list: / / returnersfind keywords # #return: mysql # #return: mysql,slack,redis # #return: # - mysql # - hipchat # - slack mysql.host: '192.168.143.103' //mysql host ip mysql.user: 'salt' mysql.pass: 'salt' //Add five lines mysql.db: 'salt' mysql.port: 3306
Store the test in mysql on the Master
[root@master salt]# salt '*' test.ping
node1:
True
[root@master salt]# salt '*' test.ping --return mysql
node1:
True
Query in database
[root@node2 ~]# mysql -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 12
Server version: 10.3.28-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> use salt
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [salt]> select * from salt_returns\G
Empty set (0.000 sec)
MariaDB [salt]> select * from salt_returns\G
*************************** 1. row ***************************
fun: test.ping
jid: 20211106152012654844 \\Viewable jid result
return: true
id: node1
success: 1
full_ret: {"success": true, "return": true, "retcode": 0, "jid": "20211106152012654844", "fun": "test.ping", "fun_args": [], "id": "node1"}
alter_time: 2021-11-06 11:20:12
1 row in set (0.000 sec)
[root@master salt]# salt-run jobs.lookup_jid 20211106152012654844
node1:
True
2. job cache
2.1 job cache process
When returning, the Minion directly interacts with the storage server. Therefore, it is necessary to install modules with specified storage methods on each Minion, such as Python mysql. Can we directly store the returned results on the Master to the storage server?
The answer is yes. This method is called job cache. It means that after Minion returns the results to the Master, the Master caches the results locally, and then stores the cached results to the specified storage server, such as mysql.
Open the master on the master side_ job_ cache
//Establish mysql service [root@master ~]# dnf -y install mariadb [root@master ~]# mysql -usalt -psalt -h192.168.143.103 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 11 Server version: 10.3.28-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> quit Bye //Note: install the MySQL Python module first [root@master jobs]# dnf list all|grep -i 'mysql' |grep 'python' python2-PyMySQL.noarch 0.8.0-10.module_el8.4.0+642+1dc4fb01 appstream python3-PyMySQL.noarch 0.10.1-2.module_el8.4.0+666+456f5f48 appstream python38-PyMySQL.noarch 0.10.1-1.module_el8.4.0+677+b84873a2 appstream python39-PyMySQL.noarch 0.10.1-2.module_el8.4.0+680+7b309a77 appstream [root@master jobs]# dnf -y install python3-PyMySQL //Configure job of mater_ cache [root@master ~]# vim /etc/salt/master . . . #job_cache: True / / search keyword job_cache mysql.host: '192.168.143.103' //mysql host ip mysql.user: 'salt' mysql.pass: 'salt' //Add six lines mysql.db: 'salt' mysql.port: 3306 master_job_cache: mysql . . . [root@master ~]# systemctl restart salt-master.service
Empty table contents in database server
[root@node2 ~]# mysql -p Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 10 Server version: 10.3.28-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> delete from salt.salt_returns; Query OK, 1 row affected (0.001 sec) MariaDB [(none)]> select * from salt.salt_returns; Empty set (0.000 sec)
Test again on the master to see if it can be stored in the database
[root@master ~]# salt '*' cmd.run 'df -h'
node1:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 876M 0 876M 0% /dev
tmpfs 896M 60K 896M 1% /dev/shm
tmpfs 896M 8.7M 887M 1% /run
tmpfs 896M 0 896M 0% /sys/fs/cgroup
/dev/mapper/cl-root 64G 2.9G 61G 5% /
/dev/sr0 9.3G 9.3G 0 100% /mnt
/dev/sda1 1014M 197M 818M 20% /boot
/dev/mapper/cl-home 32G 255M 31G 1% /home
tmpfs 180M 0 180M 0% /run/user/0
//node2 looks at the data again and finds that it has been stored
MariaDB [(none)]> select * from salt.salt_returns\G
*************************** 1. row ***************************
fun: test.ping
jid: 20211107063227111550
return: true
id: node1
success: 1
full_ret: {"cmd": "_return", "id": "node1", "success": true, "return": true, "retcode": 0, "jid": "20211107063227111550", "fun": "test.ping", "fun_args": [], "_stamp": "2021-11-07T06:32:27.338308"}
alter_time: 2021-11-07 01:32:27
1 row in set (0.000 sec)
MariaDB [(none)]> select * from salt.salt_returns\G
*************************** 1. row ***************************
fun: test.ping
jid: 20211107063227111550
return: true
id: node1
success: 1
full_ret: {"cmd": "_return", "id": "node1", "success": true, "return": true, "retcode": 0, "jid": "20211107063227111550", "fun": "test.ping", "fun_args": [], "_stamp": "2021-11-07T06:32:27.338308"}
alter_time: 2021-11-07 01:32:27
*************************** 2. row ***************************
fun: runner.jobs.lookup_jid
jid: 20211107063530824465
return: {"fun": "runner.jobs.lookup_jid", "jid": "20211107063530824465", "user": "root", "fun_args": ["20211107063227111550"], "_stamp": "2021-11-07T06:35:31.530054", "return": {"node1": true}, "success": true}
id: master_master
success: 0
full_ret: {"id": "master_master", "tgt": "master_master", "jid": "20211107063530824465", "return": {"fun": "runner.jobs.lookup_jid", "jid": "20211107063530824465", "user": "root", "fun_args": ["20211107063227111550"], "_stamp": "2021-11-07T06:35:31.530054", "return": {"node1": true}, "success": true}, "fun": "runner.jobs.lookup_jid", "user": "root"}
alter_time: 2021-11-07 01:35:47
*************************** 3. row ***************************
fun: cmd.run
jid: 20211107063957998953
return: "Filesystem Size Used Avail Use% Mounted on\ndevtmpfs 876M 0 876M 0% /dev\ntmpfs 896M 60K 896M 1% /dev/shm\ntmpfs 896M 8.7M 887M 1% /run\ntmpfs 896M 0 896M 0% /sys/fs/cgroup\n/dev/mapper/cl-root 64G 2.9G 61G 5% /\n/dev/sr0 9.3G 9.3G 0 100% /mnt\n/dev/sda1 1014M 197M 818M 20% /boot\n/dev/mapper/cl-home 32G 255M 31G 1% /home\ntmpfs 180M 0 180M 0% /run/user/0"
id: node1
success: 1
full_ret: {"cmd": "_return", "id": "node1", "success": true, "return": "Filesystem Size Used Avail Use% Mounted on\ndevtmpfs 876M 0 876M 0% /dev\ntmpfs 896M 60K 896M 1% /dev/shm\ntmpfs 896M 8.7M 887M 1% /run\ntmpfs 896M 0 896M 0% /sys/fs/cgroup\n/dev/mapper/cl-root 64G 2.9G 61G 5% /\n/dev/sr0 9.3G 9.3G 0 100% /mnt\n/dev/sda1 1014M 197M 818M 20% /boot\n/dev/mapper/cl-home 32G 255M 31G 1% /home\ntmpfs 180M 0 180M 0% /run/user/0", "retcode": 0, "jid": "20211107063957998953", "fun": "cmd.run", "fun_args": ["df -h"], "_stamp": "2021-11-07T06:39:58.175936"}
alter_time: 2021-11-07 01:39:58
3 rows in set (0.000 sec)
[root@master ~]# salt-run jobs.lookup_jid 20211107063957998953
node1:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 876M 0 876M 0% /dev
tmpfs 896M 60K 896M 1% /dev/shm
tmpfs 896M 8.7M 887M 1% /run
tmpfs 896M 0 896M 0% /sys/fs/cgroup
/dev/mapper/cl-root 64G 2.9G 61G 5% /
/dev/sr0 9.3G 9.3G 0 100% /mnt
/dev/sda1 1014M 197M 818M 20% /boot
/dev/mapper/cl-home 32G 255M 31G 1% /home
tmpfs 180M 0 180M 0% /run/user/0
Query in database
//Test trigger first
[root@master ~]# salt '*' test.ping
node1:
True
//Query whether the database records
MariaDB [(none)]> select * from salt.salt_returns\G
*************************** 1. row ***************************
fun: test.ping
jid: 20211107063227111550
return: true
id: node1
success: 1
full_ret: {"cmd": "_return", "id": "node1", "success": true, "return": true, "retcode": 0, "jid": "20211107063227111550", "fun": "test.ping", "fun_args": [], "_stamp": "2021-11-07T06:32:27.338308"}
alter_time: 2021-11-07 01:32:27
1 row in set (0.000 sec)
//jobs in the data table view the contents of the run
[root@master ~]# salt-run jobs.lookup_jid 20211107063227111550
node1:
True
2.2 job management
jobs.active view jobs information during execution
[root@node1 ~]# rpm -qa|grep httpd
[root@master ~]# salt '*' pkg.install httpd
During execution
[root@master ~]# salt-run jobs.active
20211107064709735121:
----------
Arguments:
- httpd
Function:
pkg.install
Returned:
Running:
|_
----------
node1:
1781
StartTime:
2021, Nov 07 06:47:09.735121
Target:
*
Target-type:
glob
User:
root
[root@master ~]# salt '*' pkg.install httpd
//Execution complete or error
[root@master ~]# salt-run jobs.active
//Display empty
jobs.list_jobs lists all JIDS
[root@master ~]# salt-run jobs.list_jobs
20211107063227111550:
----------
Arguments:
Function:
test.ping
StartTime:
2021, Nov 07 06:32:27.111550
Target:
*
Target-type:
glob
User:
root
20211107063530824465:
----------
Arguments:
Function:
runner.jobs.lookup_jid
StartTime:
2021, Nov 07 06:35:30.824465
Target:
master_master
Target-type:
list
User:
root
20211107063957998953:
----------
Arguments:
- df -h
Function:
cmd.run
StartTime:
2021, Nov 07 06:39:57.998953
Target:
*
Target-type:
glob
User:
root
20211107064249736251:
----------
Arguments:
Function:
runner.jobs.lookup_jid
StartTime:
2021, Nov 07 06:42:49.736251
Target:
master_master
Target-type:
list
User:
root
20211107064418404632:
----------
Arguments:
Function:
runner.jobs.active
StartTime:
2021, Nov 07 06:44:18.404632
Target:
master_master
Target-type:
list
User:
root
20211107064419142800:
----------
Arguments:
Function:
saltutil.running
StartTime:
2021, Nov 07 06:44:19.142800
Target:
*
Target-type:
glob
User:
root
20211107064709735121:
----------
Arguments:
- httpd
Function:
pkg.install
StartTime:
2021, Nov 07 06:47:09.735121
Target:
*
Target-type:
glob
User:
root
20211107064714840083:
----------
Arguments:
Function:
runner.jobs.active
StartTime:
2021, Nov 07 06:47:14.840083
Target:
master_master
Target-type:
list
User:
root
20211107064714927092:
----------
Arguments:
- 20211107064709735121
Function:
saltutil.find_job
StartTime:
2021, Nov 07 06:47:14.927092
Target:
- node1
Target-type:
list
User:
root
20211107064716552160:
----------
Arguments:
Function:
saltutil.running
StartTime:
2021, Nov 07 06:47:16.552160
Target:
*
Target-type:
glob
User:
root
20211107064732729071:
----------
Arguments:
Function:
runner.jobs.active
StartTime:
2021, Nov 07 06:47:32.729071
Target:
master_master
Target-type:
list
User:
root
20211107064733403300:
----------
Arguments:
Function:
saltutil.running
StartTime:
2021, Nov 07 06:47:33.403300
Target:
*
Target-type:
glob
User:
root
20211107065210668753:
----------
Arguments:
Function:
runner.jobs.lookup_jid
StartTime:
2021, Nov 07 06:52:10.668753
Target:
master_master
Target-type:
list
User:
root
-v get the jid of the task
[root@master ~]# salt '*' cmd.run 'df -h' -v
Executing job with jid 20211107065531741047
-------------------------------------------
node1:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 876M 0 876M 0% /dev
tmpfs 896M 40K 896M 1% /dev/shm
tmpfs 896M 8.7M 887M 1% /run
tmpfs 896M 0 896M 0% /sys/fs/cgroup
/dev/mapper/cl-root 64G 3.0G 61G 5% /
/dev/sr0 9.3G 9.3G 0 100% /mnt
/dev/sda1 1014M 197M 818M 20% /boot
/dev/mapper/cl-home 32G 255M 31G 1% /home
tmpfs 180M 0 180M 0% /run/user/0
jobs.lookup_jid obtains the returned result of this task through jid
[root@master ~]# salt-run jobs.lookup_jid 20211107064709735121
node1:
----------
apr:
----------
new:
1.6.3-11.el8
old:
apr-util:
----------
new:
1.6.1-6.el8
old:
apr-util-bdb:
----------
new:
1.6.1-6.el8
old:
apr-util-openssl:
----------
new:
1.6.1-6.el8
old:
centos-logos-httpd:
----------
new:
85.8-1.el8
old:
httpd:
----------
new:
2.4.37-39.module_el8.4.0+950+0577e6ac.1
old:
httpd-filesystem:
----------
new:
2.4.37-39.module_el8.4.0+950+0577e6ac.1
old:
httpd-tools:
----------
new:
2.4.37-39.module_el8.4.0+950+0577e6ac.1
old:
mailcap:
----------
new:
2.1.48-3.el8
old:
mod_http2:
----------
new:
1.15.7-3.module_el8.4.0+778+c970deab
old:
[root@master ~]# salt-run jobs.lookup_jid 20211107065531741047
node1:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 876M 0 876M 0% /dev
tmpfs 896M 40K 896M 1% /dev/shm
tmpfs 896M 8.7M 887M 1% /run
tmpfs 896M 0 896M 0% /sys/fs/cgroup
/dev/mapper/cl-root 64G 3.0G 61G 5% /
/dev/sr0 9.3G 9.3G 0 100% /mnt
/dev/sda1 1014M 197M 818M 20% /boot
/dev/mapper/cl-home 32G 255M 31G 1% /home
tmpfs 180M 0 180M 0% /run/user/0