1, rsync overview
1. The concept of rsync
rsync (Remote Sync) is an open source fast incremental backup tool, which can mirror and synchronize the entire directory tree between different hosts, support incremental backup, maintain links and permissions, and adopt optimized synchronization algorithm to perform compression before transmission. Therefore, it is not often suitable for remote backup, mirror server and other applications.
Support local replication or synchronization with other SSH and rsync hosts
Official website: http://rsync.samba.org
2. rsync sync source (backup source)
Refers to the remote server of the backup operation, also known as the backup source.
In the remote synchronization task, the client responsible for initiating rsync synchronization is called the initiator, and the server responsible for responding to rsync synchronization from the client is called the synchronization source. In the process of synchronization, the synchronization source is responsible for providing the original location of the file, and the initiator should have read permission to the location.
2, Configure rsync
1. Turn off the firewall
[root@rsync ~]# systemctl stop firewalld [root@rsync ~]# systemctl disable firewalld [root@rsync ~]# setenforce 0 setenforce: SELinux is disabled
2. Check whether rsync is installed (installed by default)
rpm -q rsync #rsyn is installed by default in general system
3. Configure the / etc/rsync.conf configuration file
[root@rsync ~]# vim /etc/rsyncd.conf ##Add the following configuration uid = nobody gid = nobody ##Locked in the source directory use chroot = yes ##Listening address address = 192.168.100.6 ##The listening port TCP / UDP 873 can be viewed through "cat /etc/services | grep rsync" port 873 ##Log file location log file = /var/log/rsyncd.log ##The file location where the process ID is stored pid file = /var/run/rsyncd.pid ##Allowed client addresses hosts allow = 192.168.100.0/24 ##Shared module ##Shared module name [qiao] ##The actual path to the source directory path = /var/www/html comment = Document Root of www.qiao.com ##Is it read-only read only = yes ##File types that are no longer compressed during synchronization dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z ##Authorized accounts. Multiple accounts are separated by spaces auth users = backuper ##Data file for storing account information secrets file = /etc/rsyncd_users.db
If anonymity is adopted, just remove the "auth users" and "secrets file" configurations.
4. Create a data file for the backup account
vim /etc/rsyncd_users.db backuper:abc123 //There is no need to establish a system user with the same name chmod 600 /etc/rsyncd_users.db //SSH -i key file location root@192.168.200.1 #The permission to authorize the remote login # key file needs to be 600
5. Ensure that all users have access to the source directory / var/www/html
The httpd service needs to be installed
yum -y install httpd systemctl start httpd systemctl enable httpd chmod +r /var/www/html/ ls -ld /var/www/html/ #Show file directory permissions in long format drwxr-xr-x. 2 root root 6 2 September 28:01 /var/www/html
6. Start rsync service
rsync --daemon //Start the rsync service and run it as an independent listening service (daemon) netstat -anpt | grep rsync //Method of closing rsync service kill $(cat /var/run/rsyncd.pid) rm -rf /var/run/rsyncd.pid
7. Prepare test web pages
[root@rsync ~]# echo 'this is rsync test!' > /var/www/html/test.html
3, Basic usage of rsync command
Syntax: rsync [options] original location destination location
Common options:
| option | explain |
|---|---|
| -r | Recursive mode, including all files in directories and subdirectories. |
| -l | For symbolic link files, they are still copied as symbolic link files. |
| -v | Displays verbose information about the synchronization process. |
| -z | compress when transferring files. |
| -a | The archive mode preserves the permissions, attributes and other information of the file, which is equivalent to the combined option "- rlptgoD" |
| -p | Keep the permission mark of the file. |
| -t | Retain the time stamp of the file. |
| -g | Keep the belonging group flag of the file (for super users only). |
| -o | Keep the master tag of the file (for super users only). |
| -H | Keep hard linked files. |
| -A | Retain ACL attribute information. |
| -D | Keep equipment documents and other special documents. |
| –delete | Delete files that exist in the destination location but not in the original location. |
4, Initiator configuration
1. Turn off the firewall
[root@backuper ~]# systemctl stop firewalld [root@backuper ~]# systemctl disable firewalld [root@backuper ~]# setenforce 0
2. Check whether rsync is installed
rpm -q rsync //rsync is installed by default in general system
3. Download the specified resources to the local / opt directory for backup
Format I:
rsync -avz backuper@192.168.100.6::qiao /opt/ //Enter the password after entering
Format 2:
rsync -avz rsync://backuper@192.168.100.6/qiao /opt // Enter the password after entering
4. Interactive free format configuration
[root@backuper ~]# echo "666520" > /etc/server.pass [root@backuper ~]# chmod 600 /etc/server.pass [root@backuper ~]# rm -rf /var/www/html/* [root@backuper ~]# rsync -avz --password-file=/etc/server.passbackuper@192.168.100.6::qiao /var/www/html
5. Scheduled timing synchronization
[root@backuper ~]# crontab -e 0 6 * * * /usr/bin/rsync/ -az --delete --password-file=/etc/server.pass backuper@192.168.100.6::qiao /var/www/html [root@backuper ~]# systemctl restart crond [root@backuper ~]# systemctl enable crond // Start and start scheduled tasks
5, inotify+rsync real-time synchronization
1. Overview of inotify + Rsync real-time synchronization
• inotify notification interface can be used to monitor various changes of file system, such as file access, deletion, movement, modification, etc. Using this mechanism, it is very convenient to realize file change alarm and incremental backup, and respond to changes in directories or files in time
• combining inotify mechanism with rsync tool can realize triggered backup (real-time synchronization), that is, as long as the document in the original location changes, the incremental backup operation will be started immediately; Otherwise, it is in a silent waiting state
• because the inotify notification mechanism is provided by the Linux kernel, it is mainly used for local monitoring, which is more suitable for uplink synchronization when applied in triggered backup
2. Lack of periodic synchronization
The backup time is fixed, with obvious delay and poor real-time performance
When the synchronization source does not change for a long time, intensive periodic tasks are unnecessary
3. Advantages of real-time synchronization
Once the original synchronization changes, start the backup immediately
As long as there is no change in the original synchronization, the backup is not performed
4. inotify mechanism of Linux kernel
Available from version 2.6.13
You can monitor file system changes and respond to notifications
Auxiliary software: inotify tools
5. Configure rsync+inotify at the initiator
The inotiify notification interface can be used to monitor various changes in the file system, such as file access, deletion, movement, modification, etc. Using this mechanism, it is very convenient to realize file change alarm and incremental backup, and respond to changes in directories or files in time.
Combining inotify mechanism with rsync tool can realize triggered backup (real-time synchronization), that is, as long as the document in the original location changes, the incremental backup operation will be started immediately; Otherwise, it is in a silent waiting state. In this way, the problems of delay and dense cycle when backing up according to a fixed cycle are avoided.
Because the inotify notification mechanism is provided by the Linux kernel, it is mainly used for local monitoring, which is more suitable for uplink synchronization when applied in triggered backup.
5.1 modifying the rsync source server configuration file
[root@rsync ~]# vim /etc/rsyncd.conf
##Modify read only to no
read only = no
[root@rsync ~]# kill $(cat /var/run/rsyncd.pid)
[root@rsync ~]# rm -rf /var/run/rsyncd.pid
[root@rsync ~]# rsync --daemon
[root@rsync ~]# netstat -natp | grep rsync
tcp 0 0 192.168.100.6:873 0.0.0.0:* LISTEN 1694/rsync
[root@rsync ~]# chmod 777 /var/www/html
5.2 adjusting inotify kernel parameters
In the Linux kernel, the default inotify mechanism provides three control parameters:
max_queued_events (monitor the event queue, the default value is 16384)
max_user_instances (the maximum number of monitored instances is 128 by default)
max_user_watches (the maximum number of monitoring files per instance, the default value is 8192).
When the number of directories and files to be monitored is large or changes frequently, it is recommended to increase the values of these three parameters.
cat /proc/sys/fs/inotify/max_queued_events #Monitoring event queue cat /proc/sys/fs/inotify/max_user_instances #Maximum number of monitoring instances cat /proc/sys/fs/inotify/max_user_watches #Maximum number of monitoring files per instance vim /etc/sysctl.conf #Increase each parameter fs.inotify.max_queued_events = 16384 fs.inotify.max_user_instances = 1024 fs.inotify.max_user_watches = 1048576 sysctl -p //Make kernel parameters take effect immediately
5.3 installing inotify tools
Inotify tools also need to be installed to use inotify mechanism to provide inotifywait and inotifywatch auxiliary tools to monitor and summarize changes.
inotifywait: it can monitor various events such as modify, create, move, delete and attrib, and output the results immediately once the changes are made.
inotifywatch: it can be used to collect file system changes and output the summarized changes after running results.
| inotifywait common options | explain |
|---|---|
| -e | Used to specify which events to monitor |
| -m | Indicates continuous monitoring |
| -r | Represents a recursive entire directory |
| -q | Simplified output information |
Initiator configuration:
[root@backuper ~]# cd /opt [root@backuper opt]# rz -E #Import inotify tools installation package rz waiting to receive. [root@backuper opt]# tar zxvf inotify-tools-3.14.tar.gz -C /opt [root@backuper opt]# cd inotify-tools-3.14/ [root@backuper inotify-tools-3.14]# ./configure [root@backuper inotify-tools-3.14]# make -j 2 && make install
Open a new window connection at the initiator, and then operate to view it in the original window
New terminal operation:
[root@backuper ~]# cd /var/www/html [root@backuper html]# touch test.php [root@backuper html]# mv test.php test.txt [root@backuper html]# echo 'test' > test.txt [root@backuper html]# rm -rf test.txt
Original terminal monitoring:
[root@backuper inotify-tools-3.14]# inotifywait -mrq -e modify,create,move,delete /var/www/html /var/www/html/ CREATE test.php /var/www/html/ MOVED_FROM test.php /var/www/html/ MOVED_TO test.txt /var/www/html/ MODIFY test.txt /var/www/html/ DELETE test.txt
5.4 write trigger synchronization script at another terminal
[root@backuper html]# vim /opt/inotify.sh
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html/"
RSYNC_CMD="rsync -azH --delete --password-file=/etc/server.pass /var/www/html/ backuper@192.168.100.6::qiao/"
$INOTIFY_CMD | while read DIRECTORY EVENT FILE
##while determines whether the monitoring record is received
do
if [ $(pgrep rsync | wc -l) -le 0 ] ; then
$RSYNC_CMD
fi
done
[root@backuper opt]# chmod +x inotify.sh
[root@backuper opt]# chmod 777 /var/www/html/
[root@backuper opt]# chmod +x /etc/rc.d/rc.local
[root@backuper opt]# echo '/opt/inotify.sh' >> /etc/rc.d/rc.local
//Join boot auto execute
The above script is used to detect changes in the local / var/www/html directory. Once an update triggers rsync synchronization, upload and backup to the qiao shared directory of server 192.168.100.6.
5.5 verification process
The verification process of triggered uplink synchronization is as follows:
(1) Run / opt / inotify locally_ Rsync.sh script
./inotify.sh &
(2) Switch to the local / var/www/html directory and perform operations such as adding, deleting and modifying files
[root@backuper opt]# cd /var/www/html [root@backuper html]# rm -rf * [root@backuper html]# touch qiao.html [root@backuper html]# echo 'this is inotify_rsync test!' > test.html
(3) View the changes in the rsync directory in the remote server
[root@rsync ~]# cd /var/www/html [root@rsync html]# ls qiao.html [root@rsync html]# cat qiao.html this is inotify_rsync test!
6, Using rsync to quickly delete a large number of files
If you want to delete a large number of files under linux, such as 1 million and 10 million, such as / usr/local/nginx/proxy_temp's nginx cache, etc., then "rm -rf *" may not work because it takes a long time to wait. In this case, we can use rsync's – delete to handle it skillfully. rsync actually uses the substitution principle.
1. Create garbage file
[root@rsync html]#touch {1..999}.txt
2. Create an empty folder
mkdir /opt/blank
3. Delete the target directory with rsync
rsync --delete-before -a -H -v --progress --stats /opt/blank/ /var/www/html/
Option description
– delete before: the receiver deletes before transmission
-a: Archive mode, which means that files are transferred recursively and all file attributes are maintained
-H: Keep hard linked files
-v: Detailed output mode
– progress: display the transmission process in the transmission room
– stats: gives the transfer status of some files