1. Introduction to Rsync
rsync is a data image backup tool under linux system. Using the fast incremental backup tool Remote Sync, you can synchronize remotely, support local replication, or synchronize with other SSH and rsync hosts.
2. rsync features
rsync supports many features:
1. The entire directory tree and file system can be mirrored and saved;
2. It is easy to maintain the permissions, time, soft and hard links of the original documents;
3. It can be installed without special permission;
4. Optimized process, high file transmission efficiency;
5. You can use rsh, ssh and other methods to transfer files. Of course, you can also connect directly through socket;
6. Support anonymous transmission
# 3. ssh authentication protocol of Rsync rsync Command to synchronize system files remote Host authentication. There are two protocols used in the authentication process:
- ssh protocol
- rsync protocol
rsync server The terminal does not need to be started rsync of daemon Process, just get remote host The user name and password can be used directly rsync synchronize files rsync server Because the terminal does not need to be started daemon Process, so there is no configuration file/etc/rsyncd.conf
The principle of ssh authentication protocol is the same as that of scp. If you don't want to enter a password during synchronization, use ssh keygen - t RSA to configure password free
//This method omits - e ssh by default, which is equivalent to the following:
[root@localhost ~]# rsync -avz anaconda-ks.cfg -e ssh root@192.168.216.200:/tmp/
-a //The file host changes and the timestamp remains unchanged
-z //Compressed data transmission
//When the port needs to be modified, we can:
[root@localhost ~]# rsync -avz anaconda-ks.cfg -e "ssh -p2222" root@192.168.216.200:/tmp/
//The ssh protocol port is modified. The default is 22
4.rsync command options
-a --Parameter, equivalent to-rlptgoD， -r --It's recursion -l --Is a linked file, which means copying a linked file -i --list rsync Files in the server -p --Means to keep the original permissions of the file -t --Keep file original time -g --Keep the original user group of the file -o --Keep the original owner of the file -D --Equivalent to block device file -z --Compression during transmission -P --Transmission progress -v --Information such as progress during transmission, and-P It doesn't matter -q --silent mode
-
Common command formats:
- rsync [OPTION]... SRC DEST
//Copy local files. This working mode is started when neither SRC nor DES path information contains a single colon ":" separator. - rsync [OPTION]... SRC [USER@]HOST:DEST
//A remote shell program (such as rsh and ssh) is used to copy the content of the local machine to the remote machine. When DST path address packet
This mode starts when there is a single colon ':' separator. - rsync [OPTION]... [USER@]HOST:SRC DEST
//A remote shell program (such as rsh and ssh) is used to copy the content of the remote machine to the local machine. When SRC address path
This mode starts when a single colon ':' separator is included.
- rsync [OPTION]... SRC DEST
5. rsync+inotify
Compared with the traditional cp and tar backup methods, rsync has the advantages of high security, fast backup and supporting incremental backup. rsync can solve the data backup requirements with low real-time requirements, such as regularly backing up the file server data to the remote server, regularly mirroring the local disk, etc.
With the continuous expansion of the scale of the application system, there are better requirements for the security and reliability of data. rsync has gradually exposed many deficiencies in the high-end business system. First, when rsync synchronizes data, it needs to scan all files for comparison and differential transmission. If the number of files reaches the order of millions or even tens of millions, scanning all files will be very time-consuming. And what is changing is often a small part of it, which is a very inefficient way. Secondly, rsync can't monitor and synchronize data in real time. Although it can trigger synchronization through the linux daemon, there must be a time difference between the two trigger actions, which may lead to inconsistency between the server and client data and can't completely recover the data in case of application failure. For the above reasons, the rsync+inotify combination appears!
Inotify is a powerful, fine-grained and asynchronous file system event monitoring mechanism. Since 2.6.13, the linux kernel has added inotify support. Inotify can monitor various subtle events such as addition, deletion, modification and movement in the file system. Using this kernel interface, third-party software can monitor various changes of files in the file system, Inotify tools is such a third-party software.
As mentioned earlier, rsync can realize triggered file synchronization, but triggered by crontab daemon, the synchronized data will be different from the actual data. inotify can monitor various changes in the file system and trigger rsync synchronization when there is any change in the file, which just solves the real-time problem of synchronized data.
6. Practical operation
Source server: 192.168.216.200
Target server: 192.168.216.201
Requirement: synchronize / etc to the target server / tmp directory in real time
//Two end server Turn off firewalls and selinux [root@localhost ~]# systemctl disable --now firewalld [root@localhost ~]# sed -ri 's/^(SELINUX=).*/\1disabled/g' /etc/selinux/config
//Target server //Download rsync [root@localhost ~]# yum -y install rsync //Set rsyncd.conf configuration file cat >> /etc/rsyncd.conf <<EOF log file = /var/log/rsyncd.log # Log file location. This file will be generated automatically after rsync is started. There is no need to create it in advance pidfile = /var/run/rsyncd.pid # Storage location of pid files lock file = /var/run/rsync.lock # Lock files that support the max connections parameter secrets file = /etc/rsync.pass # User authentication profile, which stores user name and password, must be created manually [etc_from_client] # Custom sync name path = /lhj/ # rsync server data storage path, and client data will be synchronized to this directory comment = sync etc from client uid = root # Set rsync running permission to root gid = root # Set rsync running permission to root port = 873 # Default port ignore errors # Indicates that an error has occurred. Ignore the error use chroot = no # The default value is true and modified to no. the soft connection backup of directory files is added read only = no # Set the rsync server to read / write permission list = no # The rsync server resource list is not displayed max connections = 200 # maximum connection timeout = 600 # Set timeout auth users = admin # Multiple user names for data synchronization can be set, separated by commas in English hosts allow = 192.168.216.200 # Multiple IP addresses of clients that allow data synchronization can be set, separated by commas in English hosts deny = 192.168.1.1 # Multiple IP addresses of clients that prohibit data synchronization can be set, separated by commas in English EOF
//Create user authentication file [root@localhost ~]# echo 'admin:123456' > /etc/rsync.pass //set files permissions [root@localhost ~]# chmod 600 /etc/rsync* [root@localhost ~]# ll /etc/rsync* -rw-------. 1 root root 1860 10 November 20:25 /etc/rsyncd.conf -rw-------. 1 root root 13 10 November 20:26 /etc/rsync.pass //Start the rsync service and set the startup self startup [root@localhost ~]# yum -y install rsync-daemon.noarch [root@localhost ~]# systemctl enable --now rsyncd Created symlink from /etc/systemd/system/multi-user.target.wants/rsyncd.service to /usr/lib/systemd/system/rsyncd.service. [root@localhost ~]# ss -antl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 5 *:873 *:* LISTEN 0 128 *:111 *:* LISTEN 0 5 192.168.122.1:53 *:* LISTEN 0 128 *:22 *:* LISTEN 0 128 127.0.0.1:631 *:* LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 5 :::873 :::* LISTEN 0 128 :::111 :::* LISTEN 0 128 :::22 :::* LISTEN 0 128 ::1:631 :::* LISTEN 0 100 ::1:25 :::*
//Source server //Install eprl source [root@localhost ~]# yum -y install epel-release //To install rsync server software, you only need to install, do not start, and do not need to configure [root@localhost ~]# yum -y install rsync //Create authentication password file [root@localhost ~]# echo '123456' > /etc/rsync.pass //Set file permissions. Only the file owner has read and write permissions [root@localhost ~]# ll /etc/rsync.pass -rw-------. 1 root root 7 Oct 12 11:32 /etc/rsync.pass //Can the experiment be transmitted normally [root@localhost ~]# mkdir -pv /root/etc/123 mkdir: created directory '/root/etc' mkdir: created directory '/root/etc/123' [root@localhost ~]# rsync -avH --port 873 --progress --delete /root/etc/ admin@192.168.216.201::etc_from_client --password-file=/etc/rsync.pass
//Target server view [root@localhost ~]# ls /lhj / / this directory needs to be created in advance 123
//Install inotify tools tool to trigger rsync for synchronization in real time //Check whether the server kernel supports inotify [root@localhost ~]# ll /proc/sys/fs/inotify/ total 0 -rw-r--r--. 1 root root 0 Oct 12 11:42 max_queued_events -rw-r--r--. 1 root root 0 Oct 12 11:42 max_user_instances -rw-r--r--. 1 root root 0 Oct 12 11:42 max_user_watches //If there are three files beginning with max, it means that the server kernel supports inotify //Installing inotify tools [root@localhost ~]# yum -y install make gcc gcc-c++ [root@localhost ~]# yum -y install inotify-tools
//Writing a synchronization script is the most important step. Let the script automatically detect the directory we make
//File changes, and then execute the rsync command to synchronize it to our server
[root@localhost ~]# mkdir /scripts
[root@localhost ~]# touch /scripts/inotify.sh
[root@localhost ~]# chmod 755 /scripts/inotify.sh
[root@localhost ~]# ll /scripts/inotify.sh
-rwxr-xr-x. 1 root root 0 Oct 12 11:47 /scripts/inotify.sh
[root@localhost ~]# vim /scripts/inotify.sh
host=192.168.216.201 # IP of the target server (backup server)
src=/etc # The backup directory to be monitored on the source server (you can customize it here, but make sure it exists)
des=etc_from_client # The customized module name must be consistent with the synchronization name defined on the target server
password=/etc/rsync.pass # Password file to perform data synchronization
user=admin # User name to perform data synchronization
inotifywait=/usr/bin/inotifywait
$inotifywait -mrq --timefmt '%Y%m%d %H:%M' --format '%T %w%f%e' -e modify,delete,create,attrib $src \
| while read files;do
rsync -avzP --delete --timeout=100 --password-file=${password} $src $user@$host::$des
echo "${files} was rsynced" >>/tmp/rsync.log 2>&1
done
//Startup script
[root@localhost scripts]# nohup bash /scripts/inotify.sh &
[1] 193030
[root@localhost scripts]# nohup: ignoring input and appending output to 'nohup.out'
[root@localhost scripts]# ps -ef|grep inotify
root 211228 1713 0 12:20 pts/0 00:00:00 grep --color=auto inotify
root 2937 2936 0 12:20 pts/1 00:00:00 /usr/bin/inotifywait -mrq --timefmt %Y%m%d %H:%M --format %T %w%f%e -e modify,delete,create,attrib /etc
//Generate a new file on the source server to trigger
[root@localhost ~]# touch /etc/123
//View logs generated by inotify
[root@localhost ~]# tail /tmp/rsync.log
20211012 12:21 /etc/123CREATE was rsynced
20211012 12:21 /etc/123ATTRIB was rsynced
//Set script startup to start automatically
[root@localhost ~]# chmod +x /etc/rc.d/rc.local
[root@localhost ~]# ll /etc/rc.d/rc.local
-rwxr-xr-x. 1 root root 474 10 December 2021 /etc/rc.d/rc.local
[root@localhost ~]# echo 'nohup /bin/bash /scripts/inotify.sh' >> /etc/rc.d/rc.local
Authentication to target server
[root@localhost tmp]# cd etc/ [root@localhost etc]# ls 123 cups GREP_COLORS libssh oddjobd.conf.d rc.local sudoers cupshelpers groff libuser.conf openldap rdma sudoers.d aliases dbus-1 group libvirt opt redhat-access-insights sudo-ldap.conf alsa dconf group- locale.conf os-release redhat-release swid alternatives default grub2.cfg localtime ostree