rsync backup file in linux
How to back up files
Backup method:
cp: native replication
scp: remote replication
PUSH(Local upload to remote server): scp 1.txt root@ip:[route] [root@m01 ~]# scp 1.txt root@172.16.1.41:/opt/ PULL(Download remote server files locally): scp root@ip:[route] ./ [root@m01 ~]# scp root@172.16.1.41:/opt/2.txt ./
Disadvantages: only full copy is supported (that is, if a file is modified, you need to copy it all to update the information, which is a waste of time)
Introduction to rsync (supports incremental replication)
rsync is called remote synchronization in English. It can be seen from the name of the software that rsync has the functions of fast data replication, synchronous mirroring and remote backup between local and remote hosts. This function is similar to the scp command with ssh, but better than the scp command. scp is copied in full every time, while rsync can be copied in increments. Of course, rsync can also replicate data between different partitions or directories of the local host in full and incremental amounts, which is similar to the cp command. But it is also better than the cp command. cp is a full copy every time, while rsync can copy incrementally.
rsync official address: https://rsync.samba.org/
rsync listening port: 873
rsync operation mode: C / s client / server
rsync is called remote synchronization for short. It can realize data synchronization between different hosts, and also supports full and incremental synchronization
rsync feature
- Support copying special files, such as connection files, devices, etc.
- It can exclude the synchronization of specified files or directories, which is equivalent to the exclusion function of the packaging command tar.
- You can keep all attributes of the original file or directory, such as permission, time, soft and hard links, owner and group, unchanged – p.
- Incremental synchronization can be realized, that is, only the changed data can be synchronized, so the data transmission efficiency is very high (tar-N).
- You can use rcp, rsh, ssh and other methods to transfer files (rsync itself does not encrypt data).
- You can transfer files and data (server and client) * * * * * through socket (process mode).
- Support anonymous live authentication (without system users) process mode transmission, which can realize convenient and safe data backup and mirroring.
rsync transmission mode
Similar to scp, rsync also has two transmission modes: push and pull
push: the client pushes data locally to the server
pull: the client pulls data from the server to the local server
rsync transport mode
- Local mode (similar to cp, push and pull are not supported, but simple replication)
- Remote mode (similar to scp, but different from scp). scp only supports full backup, and rsync supports incremental backup and differential backup
- Daemon mode (client and server)
rsync usage
Format: rsync parameter source file path root@ip : destination file path
parameter | effect |
---|---|
-a | Archive mode transfer, equal to - tropgdl - t - R - O - P - G - D - L |
-v | Detailed mode output, print rate, number of files, etc |
-z | Compression during transmission to improve efficiency |
-r | Recursive transmission of directories and subdirectories, that is, all directories under the directory are transmitted the same |
-t | Keep file time information |
-o | Keep file master information |
-g | Keep file group information |
-p | Keep file permissions |
-l | Keep soft connection |
-P | Display information such as synchronization process and transmission progress |
-D | Keep device file information |
-L | Keep the target file pointed to by the soft connection |
-e | Using the channel protocol, specify the shell program that replaces rsh |
--append | The specified file continues to be transferred where the last transfer was interrupted |
--append-verify | Use the parameter to continue transmission (after the breakpoint continues transmission, verify the file, and repair the file if it is different) |
--exclude = file name | Specifies to exclude files that do not need to be transferred |
--Exclude from = file name | Exclude file names written in the file (multiple file names can be written in the file) |
--bwlimit = transmission speed | Speed limit transmission (unit: MB) |
--delete | Keep the target directory and source directory data consistent |
--Password file = file name | Use password file (password should be written in the file) |
--port | Specify port transport |
eg:
-t #Keep file time information [root@m01 ~]# rsync -vzrt ./a/b/c/2.txt root@172.16.1.41:/opt/ -o #Keep file master information -g #Keep file group information [root@m01 ~]# rsync -vzrtgo ./a/b/c/2.txt root@172.16.1.41:/opt/ -p #Keep file permissions [root@m01 ~]# rsync -vzrtgop ./a/b/c/2.txt root@172.16.1.41:/opt/ -l #Keep soft connection [root@m01 ~]# rsync -vzrtgopl ./* root@172.16.1.41:/opt/ -P #Display information such as synchronization process and transmission progress [root@m01 ~]# rsync -vzrtgoplP ./* root@172.16.1.41:/opt/ -D #Keep device file information [root@m01 dev]# rsync -vzrtgDopl /dev/tty1 root@172.16.1.41:/opt/ -L #Keep the target file pointed to by the soft connection -e #Using the channel protocol, specify the shell program that replaces rsh --bwlimit=100 # Speed limit transmission (unit: MB) [root@m01 ~]# rsync -avzP --append-verify --bwlimit=10 ./* root@172.16.1.41:/opt/
-v #Detailed mode output, print rate, number of files, etc [root@m01 ~]# rsync -v ./2.txt root@172.16.1.41:/opt/
-r #Recursive transmission of directories and subdirectories, that is, all directories under the directory are transmitted the same. [root@m01 ~]# rsync -vzr ./a root@172.16.1.41:/opt/
rsync incremental replication
-a #Archive mode transfer, equal to - tropgdl - t - R - O - P - G - D - L -z #Compression during transmission to improve efficiency [root@m01 ~]# rsync -vz ./2.txt root@172.16.1.41:/opt/
--append # The specified file continues to be transferred where the last transfer was interrupted [root@m01 ~]# rsync -avzP --append ./* root@172.16.1.41:/opt/
--append-verify # Use the parameter to continue transmission (after the breakpoint continues transmission, verify the file, and repair the file if it is different) [root@m01 ~]# rsync -avzP --append-verify ./* root@172.16.1.41:/opt/
--exclude=PATTERN # Specifies to exclude files that do not need to be transferred [root@m01 ~]# rsync -avzP --append-verify --exclude=2.txt ./* root@172.16.1.41:/opt/
--exclude-from=file # Exclude as specified in the file [root@m01 ~]# rsync -avzP --append-verify --exclude-from=/tmp/exclude.txt ./* root@172.16.1.41:/opt/
rsync daemon mode
Server
Follow these steps to set up
1,install [root@backup ~]# yum install -y rsync 2,Modify profile [root@m01 ~]# vim /etc/rsyncd.conf uid = rsync gid = rsync port = 873 fake super = yes use chroot = no max connections = 200 timeout = 600 ignore errors read only = false list = false auth users = rsync_backup secrets file = /etc/rsync.passwd log file = /var/log/rsyncd.log ##################################### [backup] # Write the module name casually comment = welcome to backup! path = /backup [linux] # Write the module name casually comment = welcome to linux! path=/tmp/linux 3,Create system user [root@backup opt]# groupadd rsync -g 666 # Write freely in the group [root@backup opt]# useradd rsync -u 666 -g 666 -M -s /sbin/nologin -r # The owner writes casually 4,Create password file [root@backup opt]# echo "rsync_backup:123456" > /etc/rsync.passwd # Set the password yourself 5,Authorization (must be 600) [root@backup opt]# chmod 600 /etc/rsync.passwd 6,Create backup directory [root@backup opt]# mkdir /backup [root@backup opt]# mkdir /tmp/linux 7,Directory authorization [root@backup opt]# chown rsync.rsync /backup/ [root@backup opt]# chown rsync.rsync /tmp/linux/ 8,Turn off firewalls and selinux [root@backup opt]# systemctl disable --now firewalld [root@backup opt]# setenforce 0 9,start-up rsyncd service [root@backup opt]# systemctl start rsyncd
client
Follow these steps to set up
Method 1: enter your own password [root@m01 ~]# rsync -avzP ./* rsync_backup@172.16.1.41::backup rsync_backup : Virtual user, used only during data transmission 172.16.1.41 : backup Server side IP backup : Module name Method 2: set the password file and read it at run time 1,Write password file [root@backup opt]# echo "123456" > /etc/rsync.passwd 2,to grant authorization [root@m01 ~]# chmod 600 /etc/rsync.passwd 3,connect [root@m01 ~]# rsync -avzP --password-file=/etc/rsync.passwd ./* rsync_backup@172.16.1.41::linux Method 3: add environment variables 1,Define environment variables export RSYNC_PASSWORD=123456 2,synchronization [root@m01 ~]# rsync -avzP ./* rsync_backup@172.16.1.41::linux
rsync real-time synchronization
rsync does not support real-time synchronization. Usually, we use inotify software to monitor file changes in real time. Once inotify monitors file changes, we immediately call rsync for synchronization.
parameter | effect |
---|---|
-m | Continuous monitoring |
-r | recursion |
-q | Silent, printing only time information |
--timefmt | Specifies the output time format |
--format | Specify event output format (% Xe event,% w directory,% f file) |
-e | Specify the monitored events (access access, modify content, attrib attribute, close_write, open, create, delete) |
Format:
[root@m01 ~]# /usr/bin/inotifywait -mrq --format '%Xe %w %f' -e create,modify,delete,attrib,close_write /root # Continuously monitor the creation, modification, deletion, attribute modification and file content in the local / root directory
Real time monitoring and synchronization
[root@m01 ~]# /usr/bin/inotifywait -mrq --format '%Xe %w %f' -e create,modify,delete,attrib,close_write /root | while read line;do cd /root rsync -avzP --delete --password-file=/etc/rsync.passwd ./* rsync_backup@172.16.1.41::linux done # Because the root directory is monitored, switch to / root directory