rsync synchronization service

1. Introduction to Rsync

rsync is a data image backup tool under linux system. Using the fast incremental backup tool Remote Sync, you can synchronize remotely, support local replication, or synchronize with other SSH and rsync hosts.

2. rsync features

rsync supports many features:

1. The entire directory tree and file system can be mirrored and saved;
2. It is easy to maintain the permissions, time, soft and hard links of the original documents;
3. It can be installed without special permission;
4. Optimized process, high file transmission efficiency;
5. You can use rsh, ssh and other methods to transfer files. Of course, you can also connect directly through socket;
6. Support anonymous transmission

# 3. ssh authentication protocol of Rsync
rsync Command to synchronize system files remote Host authentication. There are two protocols used in the authentication process:
  • ssh protocol
  • rsync protocol
rsync server The terminal does not need to be started rsync of daemon Process, just get remote host The user name and password can be used directly rsync synchronize files
rsync server Because the terminal does not need to be started daemon Process, so there is no configuration file/etc/rsyncd.conf

The principle of ssh authentication protocol is the same as that of scp. If you don't want to enter a password during synchronization, use ssh keygen - t RSA to configure password free

//This method omits - e ssh by default, which is equivalent to the following:
[root@localhost ~]# rsync -avz anaconda-ks.cfg -e ssh root@

    -a  //The file host changes and the timestamp remains unchanged
    -z  //Compressed data transmission
//When the port needs to be modified, we can:
[root@localhost ~]# rsync -avz anaconda-ks.cfg -e "ssh -p2222" root@  
//The ssh protocol port is modified. The default is 22

4.rsync command options

-a --Parameter, equivalent to-rlptgoDīŧŒ 
-r --It's recursion 
-l --Is a linked file, which means copying a linked file
-i --list rsync Files in the server
-p --Means to keep the original permissions of the file
-t --Keep file original time 
-g --Keep the original user group of the file 
-o --Keep the original owner of the file 
-D --Equivalent to block device file 
-z --Compression during transmission
-P --Transmission progress 
-v --Information such as progress during transmission, and-P It doesn't matter 
-q --silent mode 
  • Common command formats:

    • rsync [OPTION]... SRC DEST
      //Copy local files. This working mode is started when neither SRC nor DES path information contains a single colon ":" separator.
    • rsync [OPTION]... SRC [USER@]HOST:DEST
      //A remote shell program (such as rsh and ssh) is used to copy the content of the local machine to the remote machine. When DST path address packet
      This mode starts when there is a single colon ':' separator.
    • rsync [OPTION]... [USER@]HOST:SRC DEST
      //A remote shell program (such as rsh and ssh) is used to copy the content of the remote machine to the local machine. When SRC address path
      This mode starts when a single colon ':' separator is included.

5. rsync+inotify

Compared with the traditional cp and tar backup methods, rsync has the advantages of high security, fast backup and supporting incremental backup. rsync can solve the data backup requirements with low real-time requirements, such as regularly backing up the file server data to the remote server, regularly mirroring the local disk, etc.
With the continuous expansion of the scale of the application system, there are better requirements for the security and reliability of data. rsync has gradually exposed many deficiencies in the high-end business system. First, when rsync synchronizes data, it needs to scan all files for comparison and differential transmission. If the number of files reaches the order of millions or even tens of millions, scanning all files will be very time-consuming. And what is changing is often a small part of it, which is a very inefficient way. Secondly, rsync can't monitor and synchronize data in real time. Although it can trigger synchronization through the linux daemon, there must be a time difference between the two trigger actions, which may lead to inconsistency between the server and client data and can't completely recover the data in case of application failure. For the above reasons, the rsync+inotify combination appears!

Inotify is a powerful, fine-grained and asynchronous file system event monitoring mechanism. Since 2.6.13, the linux kernel has added inotify support. Inotify can monitor various subtle events such as addition, deletion, modification and movement in the file system. Using this kernel interface, third-party software can monitor various changes of files in the file system, Inotify tools is such a third-party software.
As mentioned earlier, rsync can realize triggered file synchronization, but triggered by crontab daemon, the synchronized data will be different from the actual data. inotify can monitor various changes in the file system and trigger rsync synchronization when there is any change in the file, which just solves the real-time problem of synchronized data.

6. Practical operation

Source server:
Target server:

Requirement: synchronize / etc to the target server / tmp directory in real time

//Two end server

Turn off firewalls and selinux
[root@localhost ~]# systemctl disable --now firewalld
[root@localhost ~]# sed -ri 's/^(SELINUX=).*/\1disabled/g' /etc/selinux/config

//Target server

//Download rsync
[root@localhost ~]# yum -y install rsync
//Set rsyncd.conf configuration file
 cat >> /etc/rsyncd.conf <<EOF 
log file = /var/log/rsyncd.log    # Log file location. This file will be generated automatically after rsync is started. There is no need to create it in advance
pidfile = /var/run/     # Storage location of pid files
lock file = /var/run/rsync.lock   # Lock files that support the max connections parameter
secrets file = /etc/rsync.pass    # User authentication profile, which stores user name and password, must be created manually

[etc_from_client]     # Custom sync name
path = /lhj/          # rsync server data storage path, and client data will be synchronized to this directory
comment = sync etc from client
uid = root        # Set rsync running permission to root
gid = root        # Set rsync running permission to root
port = 873        # Default port
ignore errors     # Indicates that an error has occurred. Ignore the error
use chroot = no       # The default value is true and modified to no. the soft connection backup of directory files is added
read only = no    # Set the rsync server to read / write permission
list = no     # The rsync server resource list is not displayed
max connections = 200     # maximum connection
timeout = 600     # Set timeout
auth users = admin        # Multiple user names for data synchronization can be set, separated by commas in English
hosts allow =  # Multiple IP addresses of clients that allow data synchronization can be set, separated by commas in English
hosts deny =      # Multiple IP addresses of clients that prohibit data synchronization can be set, separated by commas in English

//Create user authentication file
[root@localhost ~]# echo 'admin:123456' > /etc/rsync.pass
//set files permissions
[root@localhost ~]# chmod 600 /etc/rsync*
[root@localhost ~]# ll /etc/rsync*
-rw-------. 1 root root 1860 10 November 20:25 /etc/rsyncd.conf
-rw-------. 1 root root   13 10 November 20:26 /etc/rsync.pass
//Start the rsync service and set the startup self startup
[root@localhost ~]# yum -y install rsync-daemon.noarch
[root@localhost ~]# systemctl enable  --now rsyncd
Created symlink from /etc/systemd/system/ to /usr/lib/systemd/system/rsyncd.service.
[root@localhost ~]# ss -antl
State      Recv-Q Send-Q                    Local Address:Port                                   Peer Address:Port              
LISTEN     0      5                                     *:873                                               *:*                  
LISTEN     0      128                                   *:111                                               *:*                  
LISTEN     0      5                                                               *:*                  
LISTEN     0      128                                   *:22                                                *:*                  
LISTEN     0      128                                                                *:*                  
LISTEN     0      100                                                                 *:*                  
LISTEN     0      5                                    :::873                                              :::*                  
LISTEN     0      128                                  :::111                                              :::*                  
LISTEN     0      128                                  :::22                                               :::*                  
LISTEN     0      128                                 ::1:631                                              :::*                  
LISTEN     0      100                                 ::1:25                                               :::*                  
//Source server

//Install eprl source
[root@localhost ~]# yum -y install epel-release
//To install rsync server software, you only need to install, do not start, and do not need to configure
[root@localhost ~]# yum -y install rsync
//Create authentication password file
[root@localhost ~]# echo '123456' > /etc/rsync.pass
//Set file permissions. Only the file owner has read and write permissions
[root@localhost ~]# ll /etc/rsync.pass 
-rw-------. 1 root root 7 Oct 12 11:32 /etc/rsync.pass
//Can the experiment be transmitted normally
[root@localhost ~]# mkdir -pv /root/etc/123
mkdir: created directory '/root/etc'
mkdir: created directory '/root/etc/123'
[root@localhost ~]# rsync -avH --port 873 --progress --delete /root/etc/ admin@ --password-file=/etc/rsync.pass

//Target server view
[root@localhost ~]# ls /lhj / / this directory needs to be created in advance
//Install inotify tools tool to trigger rsync for synchronization in real time
//Check whether the server kernel supports inotify
[root@localhost ~]#  ll /proc/sys/fs/inotify/
total 0
-rw-r--r--. 1 root root 0 Oct 12 11:42 max_queued_events
-rw-r--r--. 1 root root 0 Oct 12 11:42 max_user_instances
-rw-r--r--. 1 root root 0 Oct 12 11:42 max_user_watches
//If there are three files beginning with max, it means that the server kernel supports inotify

//Installing inotify tools
[root@localhost ~]# yum -y install make gcc gcc-c++
[root@localhost ~]# yum -y install inotify-tools

//Writing a synchronization script is the most important step. Let the script automatically detect the directory we make
//File changes, and then execute the rsync command to synchronize it to our server

[root@localhost ~]# mkdir /scripts
[root@localhost ~]# touch /scripts/
[root@localhost ~]# chmod 755 /scripts/
[root@localhost ~]# ll /scripts/
-rwxr-xr-x. 1 root root 0 Oct 12 11:47 /scripts/
[root@localhost ~]# vim /scripts/
host=    # IP of the target server (backup server)
src=/etc       # The backup directory to be monitored on the source server (you can customize it here, but make sure it exists)
des=etc_from_client     # The customized module name must be consistent with the synchronization name defined on the target server
password=/etc/rsync.pass        # Password file to perform data synchronization
user=admin          # User name to perform data synchronization

$inotifywait -mrq --timefmt '%Y%m%d %H:%M' --format '%T %w%f%e' -e modify,delete,create,attrib $src \
| while read files;do
    rsync -avzP --delete  --timeout=100 --password-file=${password} $src $user@$host::$des
    echo "${files} was rsynced" >>/tmp/rsync.log 2>&1
//Startup script
[root@localhost scripts]# nohup bash /scripts/ &
[1] 193030
[root@localhost scripts]# nohup: ignoring input and appending output to 'nohup.out'

[root@localhost scripts]#  ps -ef|grep inotify
root      211228    1713  0 12:20 pts/0    00:00:00 grep --color=auto inotify
root        2937    2936  0 12:20 pts/1    00:00:00 /usr/bin/inotifywait -mrq --timefmt %Y%m%d %H:%M --format %T %w%f%e -e modify,delete,create,attrib /etc

//Generate a new file on the source server to trigger
[root@localhost ~]# touch /etc/123
//View logs generated by inotify
[root@localhost ~]# tail /tmp/rsync.log
20211012 12:21 /etc/123CREATE was rsynced
20211012 12:21 /etc/123ATTRIB was rsynced
//Set script startup to start automatically
[root@localhost ~]# chmod +x /etc/rc.d/rc.local
[root@localhost ~]#  ll /etc/rc.d/rc.local
-rwxr-xr-x. 1 root root 474 10 December 2021 /etc/rc.d/rc.local
[root@localhost ~]# echo 'nohup /bin/bash /scripts/' >> /etc/rc.d/rc.local

Authentication to target server

[root@localhost tmp]# cd etc/
[root@localhost etc]# ls
123                     cups                        GREP_COLORS      libssh                    oddjobd.conf.d  rc.local                sudoers              cupshelpers                 groff            libuser.conf              openldap        rdma                    sudoers.d
aliases                 dbus-1                      group            libvirt                   opt             redhat-access-insights  sudo-ldap.conf
alsa                    dconf                       group-           locale.conf               os-release      redhat-release          swid
alternatives            default                     grub2.cfg        localtime                 ostree    

Keywords: Linux CentOS ssh

Added by outpost on Tue, 12 Oct 2021 10:03:57 +0300