rsync backup file in linux

rsync backup file in linux

How to back up files

Backup method:

cp: native replication

scp: remote replication

PUSH(Local upload to remote server): 
	scp 1.txt root@ip:[route]
	[root@m01 ~]# scp 1.txt root@172.16.1.41:/opt/

PULL(Download remote server files locally): 
	scp root@ip:[route] ./
	[root@m01 ~]# scp root@172.16.1.41:/opt/2.txt ./

Disadvantages: only full copy is supported (that is, if a file is modified, you need to copy it all to update the information, which is a waste of time)

Introduction to rsync (supports incremental replication)

rsync is called remote synchronization in English. It can be seen from the name of the software that rsync has the functions of fast data replication, synchronous mirroring and remote backup between local and remote hosts. This function is similar to the scp command with ssh, but better than the scp command. scp is copied in full every time, while rsync can be copied in increments. Of course, rsync can also replicate data between different partitions or directories of the local host in full and incremental amounts, which is similar to the cp command. But it is also better than the cp command. cp is a full copy every time, while rsync can copy incrementally.

rsync official address: https://rsync.samba.org/
rsync listening port: 873
rsync operation mode: C / s client / server

rsync is called remote synchronization for short. It can realize data synchronization between different hosts, and also supports full and incremental synchronization

rsync feature

  1. Support copying special files, such as connection files, devices, etc.
  2. It can exclude the synchronization of specified files or directories, which is equivalent to the exclusion function of the packaging command tar.
  3. You can keep all attributes of the original file or directory, such as permission, time, soft and hard links, owner and group, unchanged – p.
  4. Incremental synchronization can be realized, that is, only the changed data can be synchronized, so the data transmission efficiency is very high (tar-N).
  5. You can use rcp, rsh, ssh and other methods to transfer files (rsync itself does not encrypt data).
  6. You can transfer files and data (server and client) * * * * * through socket (process mode).
  7. Support anonymous live authentication (without system users) process mode transmission, which can realize convenient and safe data backup and mirroring.

rsync transmission mode

Similar to scp, rsync also has two transmission modes: push and pull

push: the client pushes data locally to the server

pull: the client pulls data from the server to the local server

rsync transport mode

  • Local mode (similar to cp, push and pull are not supported, but simple replication)
  • Remote mode (similar to scp, but different from scp). scp only supports full backup, and rsync supports incremental backup and differential backup
  • Daemon mode (client and server)

rsync usage

Format: rsync parameter source file path root@ip : destination file path

parameter effect
-a Archive mode transfer, equal to - tropgdl - t - R - O - P - G - D - L
-v Detailed mode output, print rate, number of files, etc
-z Compression during transmission to improve efficiency
-r Recursive transmission of directories and subdirectories, that is, all directories under the directory are transmitted the same
-t Keep file time information
-o Keep file master information
-g Keep file group information
-p Keep file permissions
-l Keep soft connection
-P Display information such as synchronization process and transmission progress
-D Keep device file information
-L Keep the target file pointed to by the soft connection
-e Using the channel protocol, specify the shell program that replaces rsh
--append The specified file continues to be transferred where the last transfer was interrupted
--append-verify Use the parameter to continue transmission (after the breakpoint continues transmission, verify the file, and repair the file if it is different)
--exclude = file name Specifies to exclude files that do not need to be transferred
--Exclude from = file name Exclude file names written in the file (multiple file names can be written in the file)
--bwlimit = transmission speed Speed limit transmission (unit: MB)
--delete Keep the target directory and source directory data consistent
--Password file = file name Use password file (password should be written in the file)
--port Specify port transport

eg:

-t           #Keep file time information
	[root@m01 ~]# rsync -vzrt ./a/b/c/2.txt  root@172.16.1.41:/opt/
-o           #Keep file master information
-g           #Keep file group information
	[root@m01 ~]# rsync -vzrtgo  ./a/b/c/2.txt  root@172.16.1.41:/opt/
-p           #Keep file permissions
	[root@m01 ~]# rsync -vzrtgop  ./a/b/c/2.txt  root@172.16.1.41:/opt/
-l           #Keep soft connection
	[root@m01 ~]# rsync -vzrtgopl  ./*  root@172.16.1.41:/opt/
-P           #Display information such as synchronization process and transmission progress
	[root@m01 ~]# rsync -vzrtgoplP  ./*  root@172.16.1.41:/opt/
-D           #Keep device file information
	[root@m01 dev]# rsync -vzrtgDopl /dev/tty1   root@172.16.1.41:/opt/
-L           #Keep the target file pointed to by the soft connection
-e           #Using the channel protocol, specify the shell program that replaces rsh
--bwlimit=100       # Speed limit transmission (unit: MB)
	[root@m01 ~]# rsync -avzP --append-verify --bwlimit=10  ./* root@172.16.1.41:/opt/
-v           #Detailed mode output, print rate, number of files, etc
	[root@m01 ~]# rsync -v ./2.txt  root@172.16.1.41:/opt/

-r           #Recursive transmission of directories and subdirectories, that is, all directories under the directory are transmitted the same.
	[root@m01 ~]# rsync -vzr ./a  root@172.16.1.41:/opt/

rsync incremental replication

-a           #Archive mode transfer, equal to - tropgdl - t - R - O - P - G - D - L

-z           #Compression during transmission to improve efficiency
	[root@m01 ~]# rsync -vz ./2.txt  root@172.16.1.41:/opt/

--append            # The specified file continues to be transferred where the last transfer was interrupted
	[root@m01 ~]# rsync -avzP --append ./* root@172.16.1.41:/opt/

--append-verify     # Use the parameter to continue transmission (after the breakpoint continues transmission, verify the file, and repair the file if it is different)
	[root@m01 ~]# rsync -avzP --append-verify ./* root@172.16.1.41:/opt/

--exclude=PATTERN   # Specifies to exclude files that do not need to be transferred
	[root@m01 ~]# rsync -avzP --append-verify --exclude=2.txt  ./* root@172.16.1.41:/opt/

--exclude-from=file # Exclude as specified in the file
	[root@m01 ~]# rsync -avzP --append-verify --exclude-from=/tmp/exclude.txt  ./* root@172.16.1.41:/opt/

rsync daemon mode

Server

Follow these steps to set up

1,install
[root@backup ~]# yum install -y rsync

2,Modify profile
[root@m01 ~]# vim /etc/rsyncd.conf 
uid = rsync
gid = rsync
port = 873
fake super = yes
use chroot = no
max connections = 200
timeout = 600
ignore errors
read only = false
list = false
auth users = rsync_backup
secrets file = /etc/rsync.passwd
log file = /var/log/rsyncd.log
#####################################
[backup]   # Write the module name casually
comment = welcome to backup!
path = /backup
[linux]   # Write the module name casually
comment = welcome to linux!
path=/tmp/linux

3,Create system user
[root@backup opt]# groupadd rsync -g 666  # Write freely in the group
[root@backup opt]# useradd rsync -u 666 -g 666 -M -s /sbin/nologin -r  # The owner writes casually

4,Create password file
[root@backup opt]# echo "rsync_backup:123456" > /etc/rsync.passwd  # Set the password yourself

5,Authorization (must be 600)
[root@backup opt]# chmod 600 /etc/rsync.passwd

6,Create backup directory
[root@backup opt]# mkdir /backup
[root@backup opt]# mkdir /tmp/linux

7,Directory authorization
[root@backup opt]# chown rsync.rsync /backup/
[root@backup opt]# chown rsync.rsync /tmp/linux/

8,Turn off firewalls and selinux
[root@backup opt]# systemctl disable --now firewalld
[root@backup opt]# setenforce 0

9,start-up rsyncd service
[root@backup opt]# systemctl start rsyncd

client

Follow these steps to set up

Method 1: enter your own password
	[root@m01 ~]# rsync -avzP ./* rsync_backup@172.16.1.41::backup
	
	rsync_backup :  Virtual user, used only during data transmission
	172.16.1.41  :  backup Server side IP
	backup       :  Module name
	
Method 2: set the password file and read it at run time

	1,Write password file
	[root@backup opt]# echo "123456" > /etc/rsync.passwd

	2,to grant authorization
	[root@m01 ~]# chmod 600 /etc/rsync.passwd

	3,connect
	[root@m01 ~]# rsync -avzP --password-file=/etc/rsync.passwd  ./* rsync_backup@172.16.1.41::linux


Method 3: add environment variables
	1,Define environment variables
	export RSYNC_PASSWORD=123456
	
	2,synchronization
	[root@m01 ~]# rsync -avzP  ./* rsync_backup@172.16.1.41::linux

rsync real-time synchronization

rsync does not support real-time synchronization. Usually, we use inotify software to monitor file changes in real time. Once inotify monitors file changes, we immediately call rsync for synchronization.

parameter effect
-m Continuous monitoring
-r recursion
-q Silent, printing only time information
--timefmt Specifies the output time format
--format Specify event output format (% Xe event,% w directory,% f file)
-e Specify the monitored events (access access, modify content, attrib attribute, close_write, open, create, delete)

Format:

[root@m01 ~]# /usr/bin/inotifywait  -mrq  --format '%Xe  %w  %f' -e create,modify,delete,attrib,close_write  /root  # Continuously monitor the creation, modification, deletion, attribute modification and file content in the local / root directory

Real time monitoring and synchronization

[root@m01 ~]# /usr/bin/inotifywait  -mrq  --format '%Xe  %w  %f' -e create,modify,delete,attrib,close_write  /root | while read line;do
	cd  /root
	rsync -avzP --delete --password-file=/etc/rsync.passwd ./* rsync_backup@172.16.1.41::linux
done
# Because the root directory is monitored, switch to / root directory

Keywords: rsync

Added by sgoldenb on Wed, 29 Dec 2021 16:36:47 +0200