Real-time synchronous sersync combat

What is real-time synchronization

Real-time synchronization is an event that triggers a change in the current directory, which synchronizes the changed directory to a remote server.

Comparison of sersync and rsync+inotify

rsync is inevitable when it comes to data synchronization. ftp/sftp is usually used for simple server data transmission, but this method is inefficient, does not support differential incremental synchronization and does not support real-time transmission. Most people will choose rsync+inotify-tools solution for data real-time synchronization, but there are also some drawbacks in this solution (the article will specifically point out that), sersync is a tool developed by people based on the former two, which not only retains the advantages, but also strengthens real-time monitoring, file filtering and simplifies configuration. Can help users improve operational efficiency, save time and network resources.

Project address

sersync Project in Practice

1) Environmental preparation

1. Synchronize the directories of which servers in real time, then install sersync on which server

2. rsync and inotify must be installed as long as sersync is installed

backup for rsync installation

1) Install rsync service

[root@backup ~]# yum install -y rsync

2) Configuration file

[root@backup ~]# vim /etc/rsyncd.conf 
uid = www
gid = www
port = 873
fake super = yes
use chroot = no
max connections = 200
timeout = 600
ignore errors
read only = false
list = false
auth users = rsync_backup
secrets file = /etc/rsync.passwd
log file = /var/log/rsyncd.log
#####################################
[zls]
comment = welcome to oldboyedu backup!
path = /backup

[nfs]
comment = welcome to oldboyedu backup!
path = /data

3) Creating Users

[root@backup ~]# groupadd www -g 666
[root@backup ~]# useradd www -u 666 -g 666 -s /sbin/nologin -M

4) Create directories and authorize them

[root@backup ~]# mkdir /data  /backup
[root@backup ~]# chown -R www.www /data/ /backup/

5) Create password files for virtual users and authorize them

[root@backup ~]# echo 'rsync_backup:123' > /etc/rsync.passwd 
chmod 600 /etc/rsync.passwd

6) Start rsync service

[root@backup ~]# systemctl start rsyncd

NFS Service End Deployment sersync

1) Installing sersync depends on rsync and inotify

[root@nfs ~]# yum install -y rsync inotify-tools

2) Download sersync

[root@nfs ~]# wget https://raw.githubusercontent.com/wsgzao/sersync/master/sersync2.5.4_64bit_binary_stable_final.tar.gz

3) Deployment of sersync

Source package: decompression, generation, compilation and installation

Decompression:

[root@nfs ~]# tar xf sersync2.5.4_64bit_binary_stable_final.tar.gz

4) Move and rename

[root@nfs ~]# mv GNU-Linux-x86 /usr/local/sersync

5) Editing configuration files

[root@nfs ~]# vim /usr/local/sersync/confxml.xml
    <inotify>
        <delete start="true"/>
        <createFolder start="true"/>
        <createFile start="true"/>
        <closeWrite start="true"/>
        <moveFrom start="true"/>
        <moveTo start="true"/>
        <attrib start="true"/>
        <modify start="true"/>
    </inotify>
-----------------------------------------------------------------------------------------
    <sersync>
        #Monitored directory, changed to / data
        <localpath watch="/opt/tongbu">
            #Push IP (IP of backup service) 172.16.1.41, name is module name
            <remote ip="127.0.0.1" name="tongbu1"/>
            <!--<remote ip="192.168.8.39" name="tongbu"/>-->
            <!--<remote ip="192.168.8.40" name="tongbu"/>-->
        </localpath>
        <rsync>
            #Change the parameters of rsync to - az
            <commonParams params="-artuz"/>
            #Virtual user's username and password file, open authentication start = true rsync_backup/etc/rsync.pass
            <auth start="false" users="root" passwordfile="/etc/rsync.pas"/>
            <userDefinedPort start="false" port="874"/><!-- port=874 -->
            #Setting timeout time
            <timeout start="true" time="100"/><!-- timeout=100 -->
            <ssh start="false"/>
        </rsync>
        <failLog path="/tmp/rsync_fail_log.sh" timeToExecute="60"/><!--default every 60mins execute once-->
        <crontab start="false" schedule="600"><!--600mins-->
            <crontabfilter start="false">
                <exclude expression="*.php"></exclude>
                <exclude expression="info/*"></exclude>
            </crontabfilter>
        </crontab>
        <plugin start="false" name="command"/>
    </sersync>


#Full configuration file
[root@nfs ~]# cat /usr/local/sersync/confxml.xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<head version="2.5">
    <host hostip="localhost" port="8008"></host>
    <debug start="false"/>
    <fileSystem xfs="false"/>
    <filter start="false">
    <exclude expression="(.*)\.svn"></exclude>
    <exclude expression="(.*)\.gz"></exclude>
    <exclude expression="^info/*"></exclude>
    <exclude expression="^static/*"></exclude>
    </filter>
    <inotify>
    <delete start="true"/>
    <createFolder start="true"/>
    <createFile start="true"/>
    <closeWrite start="true"/>
    <moveFrom start="true"/>
    <moveTo start="true"/>
    <attrib start="true"/>
    <modify start="true"/>
    </inotify>

    <sersync>
    <localpath watch="/data">
        <remote ip="172.16.1.41" name="nfs"/>
        <!--<remote ip="192.168.8.39" name="tongbu"/>-->
        <!--<remote ip="192.168.8.40" name="tongbu"/>-->
    </localpath>
    <rsync>
        <commonParams params="-az"/>
        <auth start="true" users="rsync_backup" passwordfile="/etc/rsync.pass"/>
        <userDefinedPort start="false" port="874"/><!-- port=874 -->
        <timeout start="true" time="100"/><!-- timeout=100 -->
        <ssh start="false"/>
    </rsync>
    <failLog path="/tmp/rsync_fail_log.sh" timeToExecute="60"/><!--default every 60mins execute once-->
    <crontab start="false" schedule="600"><!--600mins-->
        <crontabfilter start="false">
        <exclude expression="*.php"></exclude>
        <exclude expression="info/*"></exclude>
        </crontabfilter>
    </crontab>
    <plugin start="false" name="command"/>
    </sersync>

    <plugin name="command">
    <param prefix="/bin/sh" suffix="" ignoreError="true"/>  <!--prefix /opt/tongbu/mmm.sh suffix-->
    <filter start="false">
        <include expression="(.*)\.php"/>
        <include expression="(.*)\.sh"/>
    </filter>
    </plugin>

    <plugin name="socket">
    <localpath watch="/opt/tongbu">
        <deshost ip="192.168.138.20" port="8009"/>
    </localpath>
    </plugin>
    <plugin name="refreshCDN">
    <localpath watch="/data0/htdocs/cms.xoyo.com/site/">
        <cdninfo domainname="ccms.chinacache.com" port="80" username="xxxx" passwd="xxxx"/>
        <sendurl base="http://pic.xoyo.com/cms"/>
        <regexurl regex="false" match="cms.xoyo.com/site([/a-zA-Z0-9]*).xoyo.com/images"/>
    </localpath>
    </plugin>
</head>

6) Create password files for virtual users and authorize them

[root@nfs sersync]# echo '123' > /etc/rsync.pass
[root@nfs sersync]# chmod 600 /etc/rsync.pass

7) View Help

[root@nfs sersync]# /usr/local/sersync/sersync2 -h
set the system param
execute: echo 50000000 > /proc/sys/fs/inotify/max_user_watches
execute: echo 327679 > /proc/sys/fs/inotify/max_queued_events
parse the command param
_______________________________________________________
Parameter - d: Enable daemon mode
 Parameter - r: Push the monitor directory and remote host once with rsync command before monitoring
 c parameter - n: Specifies the number of daemon threads opened, defaulting to 10
 Parameter - o: Specify the configuration file, using the confxml.xml file by default
 Parameter - m: Enable other modules separately, and use - M refresh CDN to open refresh CDN module
 Parameter - m: Enable other modules separately, and use - m socket to open the socket module
 Parameter - m: Enable other modules separately and use - m http to open http modules
 Without the - m parameter, the synchronization program is executed by default
________________________________________________________________

8) Start sersync

[root@nfs data]# /usr/local/sersync/sersync2 -rdo /usr/local/sersync/confxml.xml

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

1) Installation

[root@backup ~]# yum install -y rsync

2) Reconfiguration

[root@backup ~]# vim /etc/rsyncd.conf
uid = rsync
gid = rsync
port = 873
fake super = yes
use chroot = no
max connections = 200
timeout = 600
ignore errors
read only = false
list = false
auth users = rsync_backup
secrets file = /etc/rsync.passwd
log file = /var/log/rsyncd.log
#####################################
[zls]
comment = welcome to oldboyedu backup!
path = /backup

3) Creating System Users

[root@backup ~]# useradd rsync -s /sbin/nologin  -M

4) Create virtual user and password files and authorize them

[root@backup ~]# echo 'rsync_backup:123' > /etc/rsync.passwd
[root@backup ~]# chmod 600 /etc/rsync.passwd

5) Create directories and authorize them

[root@backup ~]# mkdir /backup
[root@backup ~]# chown -R rsync.rsync /backup

6) Start up service and join boot-up self-start

[root@backup ~]# systemctl start rsyncd
[root@backup ~]# systemctl enable rsyncd

-------------------------------------------------------------------------------------- rsync's client (nfs) -----------------------------------------------------------------------------------------------------------------------------------------

1) Install sersync (rsync+inotify)

[root@nfs ~]# yum install -y rsync inotify-tools

2) Install sersync

Download:

[root@nfs ~]# wget https://raw.githubusercontent.com/wsgzao/sersync/master/sersync2.5.4_64bit_binary_stable_final.tar.gz

Decompression:

[root@nfs ~]# tar xf sersync2.5.4_64bit_binary_stable_final.tar.gz

Move and rename:

[root@nfs ~]# mv GNU-Linux-x86 /usr/local/sersync

3) Modify configuration files

[root@nfs ~]# vim /usr/local/sersync/confxml.xml
    <inotify>
        <delete start="true"/>
        <createFolder start="true"/>
        <createFile start="true"/>
        <closeWrite start="true"/>
        <moveFrom start="true"/>
        <moveTo start="true"/>
        <attrib start="true"/>
        <modify start="true"/>
    </inotify>

    <sersync>
        <localpath watch="/zls">
            <remote ip="172.16.1.41" name="zls"/>
            <!--<remote ip="192.168.8.39" name="tongbu"/>-->
            <!--<remote ip="192.168.8.40" name="tongbu"/>-->
        </localpath>
        <rsync>
            <commonParams params="-az"/>
            <auth start="true" users="rsync_backup" passwordfile="/etc/rsync.pas"/>
            <userDefinedPort start="false" port="874"/><!-- port=874 -->
            <timeout start="true" time="100"/><!-- timeout=100 -->
            <ssh start="false"/>
        </rsync>

4) Create directories

[root@nfs ~]# mkdir /zls

5) Create password files and authorize them

[root@nfs ~]# echo '123' > /etc/rsync.pas
[root@nfs ~]# chmod 600 /etc/rsync.pas

6) Start sersync

[root@nfs ~]# /usr/local/sersync/sersync2 -rdo /usr/local/sersync/confxml.xml

sersync is the client of rsync

Underlying calls: rsync and inotify