rsync for remote synchronization

1, About rsync

Remote Sync
Support local replication or synchronization with other SSH and rsync hosts
Official website: http://rsync.samba.org
rsync is mainly used to synchronize data in various servers
Read only can be downloaded, not uploaded. Download depends on reading
For all contents, you only need to find the source point, and the backup point is also called the initiator
kill -9 forced end pid file still exists
kill end delete pid file

2, Usage of rsync command

- rsync  [option] Original position target position     ## Synchronize the original location to the target location

a: Archive mode, recursive retention of object attributes, equivalent to - rlptgoD if the group belongs to the master

-v: Displays verbose information about the synchronization process

-z: compress when transferring files

-H: Keep hard connection position

-A: Retain ACL attribute information

– delete: delete the files in the target location but not in the original location # and keep the files on both sides consistent (if the data on both sides is inconsistent, the content obtained by the user is inconsistent). Be careful with this command

– checksum: determines whether to skip the file based on the checksum of the object

3, rsync real-time synchronization

3.1 lack of regular synchronization

The backup time is fixed, with obvious delay and poor real-time performance
When the synchronization source does not change for a long time, intensive periodic tasks are unnecessary

3.2 advantages of real-time synchronization

Once the synchronization source changes, start the backup immediately
No backup is performed as long as the synchronization source remains unchanged

4, inotify

inotify mechanism of Linux kernel
Available from version 2.6.13
You can monitor file system changes and respond to notifications
Auxiliary software: inotify tools

4.1 adjusting inotify kernel parameters

max_queue_events: monitors the size of the event queue
max_user_instances: the maximum number of monitored instances
max_user_watches: maximum number of monitored files per instance

[root@localhost ~]# vi /etc/sysctl.conf 
......
fs.inotify.max_queued_events = 16384  ##The number of monitored files should be greater than the total number of files of the monitoring target
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576
[root@localhost ~]# sysctl -p

4.2 installing inotify tools

inotifywait: used for continuous monitoring and real-time output of results
inotifywatch: used for short-term monitoring, and the results will be issued after the task is completed
inotify is used to monitor and respond to notifications
Monitored events - modify modify create create move delete delete
Parameter description

inotifywait

Syntax:

inotifywait [-hcmrq] [-e ] [-t ] [--format ] [--timefmt ] [ ... ]

Parameters:
-h,–help
Output help information
@
Exclude files that do not need to be monitored, which can be relative paths or absolute paths.
–fromfile
Read the files to be monitored or excluded from the file, one line for each file, and the excluded files start with @.
-m, –monitor
Receive a thing without quitting and execute it indefinitely. The default behavior is to exit immediately after receiving an event.
-d, –daemon
Like – monitor, except that it runs in the background, you need to specify – outfile to output things to a file. It also means that – syslog is used.
-o, –outfile
Output things to a file instead of standard output.
-s, –syslog
Output error information to system log
-r, –recursive
Monitor all subdirectories under a directory.
-q, –quiet
Specify once and no details will be output. Specify twice and no information will be output except for fatal errors.
–exclude
Regular matching needs to exclude files, which are case sensitive.
–excludei
Regular matching needs to exclude files, ignoring case.
-t , –timeout
Set the timeout. If it is 0, it will be executed indefinitely.
-e , –event
Specify the events to monitor.
-c, –csv
Output csv format.
–timefmt
Specify the time format for the% T format in the – format option.
–format
Specifies the output format.
%w indicates the directory where the event occurred
%f indicates the file where the event occurred
%e indicates an event that occurred
%Xe events are separated by "X"
%T uses the time format defined by – timefmt

inotifywatch

Syntax:

inotifywatch

Syntax:

inotifywatch [-hvzrqf] [-e ] [-t ] [-a ] [-d ] [ ... ]

Parameters:

-h, –help
Output help information
-v, –verbose
Output details
@
Exclude files that do not need to be monitored, which can be relative paths or absolute paths.
–fromfile
Read the files to be monitored or excluded from the file, one line for each file, and the excluded files start with @.
-z, –zero
Output the rows and columns of the table, even if the element is empty
–exclude
Regular matching needs to exclude files, which are case sensitive.
–excludei
Regular matching needs to exclude files, ignoring case.
-r, –recursive
Monitor all subdirectories under a directory.
-t , –timeout
Set timeout
-e , –event
Listen only for specified events.
-a , –ascending
Sort in ascending order of the specified events.
-d , –descending
In descending order of the specified events.

Listenable events

access file reading
modify file changes.
attrib file attribute changes, such as permissions, timestamp, etc.
close_write the file opened in writable mode is closed, which does not mean that the file must have written data.
close_nowrite files opened in read-only mode are closed.
The close file is closed, no matter how it is opened.
The open file opens.
moved_to move a file or directory to the listening directory. This event is triggered even if it is moved in the same directory.
moved_ This event is triggered when a file or directory is moved out of the listening directory, even if it is moved within the same directory.
move includes moved_to and moved_from
move_ The self file or directory is removed and no longer listens to this file or directory.
create file or directory creation
delete file or directory deletion
delete_ Remove the self file or directory, and then no longer listen to this file or directory
Unmount the file system, unmount it, and then stop listening to the file system.

5, Experimental rsync remote synchronization

5.1 experimental environment

Source server 192.168.75.200
Client: 192.168.75.144

5.2 experimental purpose

The source server and client can synchronize remotely

5.3 experimental steps

Source server configuration

vim /etc/rsyncd.conf
uid = nobody
gid = nobody
use chroot = yes //Imprisoned home directory
address = 192.168.75.200
port 873
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
hosts allow = 192.168.75.0/24
[wwwroot]
path = /var/www/html
comment = www.kgc.com
read only = yes  //Read only and download, but cannot upload
dont compress   = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2  //Uncompressed files will occupy bandwidth, cause network congestion and high concurrency
auth users = ttyy
secrets file = /etc/rsyncd_users.db    //Password file
[root@rsync ~]# vim /etc/rsyncd_users.db / / write user and password
ttyy:123
~             
 yum -y install httpd
[root@rsync ~]# vim /var/www/html/index.html 
<h1>this is  yuanfuwuq webwebweb !!!!!!!!!!</h1>
rsync --daemon  //open
chmod 600 /etc/rsyncd_users.db

Initiator: 192.168.75.144

Format I:
 rsync -avz ttyy@192.168.75.200::wwwroot /opt
 Password: 123
 Format 2:
 rsync -avz ttyy@192.168.75.200::wwwroot /opt
 Password: 123
######## Interaction free########
vim /etc/server.pass
abc123
[root@localhost opt]# chmod 600 /etc/server.pass 
rsync -az --delete --password-file=/etc/server.pass ttyy@192.168.75.200::wwwroot /opt/

6, Experiment: inotify+rsync monitoring and remote synchronization

192.168.75.144 as the source server

inotify listens to local information and then synchronizes to the initiator

Source server configuration

-------—rsync+inotify---------—
Note: source side settings: 
[root@localhost html]# vim /etc/rsyncd.conf 
read only = no 
192.168.75.144 As a source server
inotify Listen for local information, and then synchronize to the initiator
 Source server: 1
[root@localhost ~]#vim /etc/sysctl.conf
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576
[root@localhost ~]# sysctl -p / / load configuration
[root@localhost ~]# yum -y install gcc gcc-c++ 
Prepare installation package  inotify-tools-3.14.tar.gz
[root@localhost ~]# tar zxvf inotify-tools-3.14.tar.gz 
[root@localhost ~]# cd inotify-tools-3.14/
[root@localhost inotify-tools-3.14]# ./configure 
[root@localhost inotify-tools-3.14]# make && make install
[root@localhost ~]# vim inotify.sh
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete  /var/www/html/"
RSYNC_CMD="rsync -azH --delete --password-file=/etc/server.pass /var/www/html/ ttyy@192.168.75.200::wwwroot "  ###ip address of source server
$INOTIFY_CMD | while read DIRECTORY EVENT FILE  ##Read the monitoring record of the output
do
        if [ $(pgrep rsync | wc -l) -le 0 ]  ##If rsync is not executing, start it immediately
	then
               		 $RSYNC_CMD
fi
done
~            
#################################Script II###########################################
 vim inotify22.sh
#!/bin/bash
dir="rsync -azH --delete --password-file=/etc/server.pass /var/www/html/ ttyy@192.168.75.200::wwwroot"
dir2="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html/"
 $dir2 | while read file
do
  $dir
done
yum -y install httpd
chmod 777 inotify22.sh
[root@localhost ~]# chmod 777 /var/www/html/  ##Both servers should be set
[root@localhost ~]# ./inotify22.sh / / in listening status, start another window to verify whether to synchronize information

Verification initiator

[root@localhost html]# touch tt4.txt
[root@localhost html]# 
Listening state 
[root@localhost ~]# ./inotify22.sh 
rsync: failed to set times on "/." (in wwwroot): Operation not permitted (1)
rsync: chgrp "/.index.html.lAX8Ak" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/.tt1.txt.Rmsur1" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/.tt2.txt.4VdQhI" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/.tt3.txt.uqdc8o" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/.tt4.txt.7MpyY5" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
rsync: failed to set times on "/." (in wwwroot): Operation not permitted (1)
rsync: chgrp "/.index.html.w1DjbA" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/.tt1.txt.eUbplh" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/.tt2.txt.WjVuvY" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/.tt3.txt.7NTAFF" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/.tt4.txt.2e0GPm" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]

Check that the initiator / var/www/html has synchronized TT4 txt
[root@rsync html]# ll
Total consumption 4
-rw-------. 1 nobody nobody 32 September 12 22:36 index html
-rw-------. 1 nobody 0 September 12 22:36 TT1 txt
-rw-------. 1 nobody 0 September 12 22:36 TT2 txt
-rw-------. 1 nobody 0 September 12 22:36 TT3 txt
-rw-------. 1 nobody 0 September 12 22:36 TT4 txt
[root@rsync html]#

Keywords: rsync

Added by guru2k9 on Fri, 14 Jan 2022 10:37:54 +0200

Programming VIP

rsync for remote synchronization

rsync for remote synchronization

1, About rsync

2, Usage of rsync command

3, rsync real-time synchronization

3.1 lack of regular synchronization

3.2 advantages of real-time synchronization

4, inotify

4.1 adjusting inotify kernel parameters

4.2 installing inotify tools

5, Experimental rsync remote synchronization

5.1 experimental environment

5.2 experimental purpose

5.3 experimental steps

6, Experiment: inotify+rsync monitoring and remote synchronization

Popular Keywords