File synchronization between linux servers (one-way synchronization: rsync+inotify, two-way synchronization: unison+inotify)

In the daily operation and maintenance work, it is necessary to ensure the high availability of files to prevent the loss of files caused by a single point. According to different scenarios, there are generally two ways of one-way and two-way synchronization.
1. Bidirectional synchronization: unison+inotify is used to start the switching scenario
2. Unidirectional synchronization: rsync+inotify} is used for manual recovery scenarios
In most cases, one-way synchronization can be used. If there is a problem, the operation and maintenance thinks that manual recovery can be used; If the real-time requirements for recovery are not high, two-way synchronization can be used. When there is a problem, the system can be switched automatically, used first, and then maintained manually, which will not affect normal use.
Let's demonstrate the construction and configuration of a two-way synchronization environment.

1 Introduction to unison

Unison is a file synchronization tool that can be used under Windows, Linux and other Unix platforms. It can keep the contents of two folders (local or network) consistent. Unison has the same features as other synchronization tools or file systems, but it also has its own characteristics:
a. Cross platform use;
b. There are no special requirements for kernel and user permissions;
c.Unison is bi-directional. It can automatically handle the non conflicting part of the two copies, and the conflicting part will be displayed for users to choose the update strategy;
d. As long as the two hosts can be connected, they can run unison. They can directly use socket connection or secure ssh connection. They do not have high requirements for bandwidth. They use a compressed transmission protocol similar to rsync.

Example environment: the environment of two CentOS 7
vm1 192.168.1.110
vm2 192.168.1.111


2. Compile and install Unison

When compiling and installing Unison through the source package under Linux, you need to use the Objective Caml compiler.

2.1 compiling and installing ocaml

[root@vm1 ~]# wget http://caml.inria.fr/pub/distrib/ocaml-4.02/ocaml-4.02.0.tar.gz 
[root@vm1 ~]# tar -xzvf ocaml-4.02.0.tar.gz
[root@vm1 ~]# cd ocaml-4.02.0
[root@vm1 ocaml-4.02.0]# ./configure
[root@vm1 ocaml-4.02.0]# make world opt
[root@vm1 ocaml-4.02.0]# make install

2.2 compiling and installing Unison

[root@vm1 ~]# wget ftp://133.31.130.35/pub/pkgsrc/distfiles/unison-2.48.3.tar.gz
[root@vm1 ~]# tar -xzvf unison-2.48.3.tar.gz
[root@vm1 ~]# cd unison-2.48.3
[root@vm1 unison-2.48.3]# make UISTYLE=text
[root@vm1 unison-2.48.3]# make install

2.3 copy untion to / usr/local/bin

[root@vm1 unison-2.48.3]# cp unison /usr/local/bin

3. Configure ssh password less login

3.3.1 synchronize ssh keygen to generate ssh login security key pair (executed separately on two machines)

ssh-keygen -t rsa

All the way back to the end
A pair of keys, ID, will be generated here_ RSA (private key file) and id_rsa.pub (public key file), saved in ~ / ssh / directory

3.2 copy public key and authorization

Will ID_ rsa. The contents of the pub (public key) file are copied to the authorized of another server_ In the keys file, complete the authorization

# Execute on vm1
cat ~/id_rsa.pub # Get the content, and then copy

# vm2 execution paste
vi ~/.ssh/authorized_keys

# vm2 file authorization
chmod 700 .ssh
chmod 600 ~/.ssh/authorized_keys

# vm2 restart SSH service
[root@vm1 ~]# service sshd restart

# vm2 test:
ssh root@vm1

The same reverse operation, mutual authorization.


4 set synchronization script

Here is an example of synchronizing nfs folders on two machines

Script on vm1 (untion. SH)

#/bin/sh
UNISON=`ps -ef |grep -v grep|grep -c inotifywait`
if [ ${UNISON} -lt 1 ]
then
    other_server="192.168.1.110"
    src1="/home/nfs/"
    dst2="/home/nfs/"
    /usr/bin/inotifywait -mrq -e create,delete,delete_self,modify,move $src1 | while read line; do
        /usr/bin/unison $src1 ssh://$other_server/$dst2
        echo -n "$line " >> /var/log/inotify.log
        echo `date | cut -d " " -f1-4` >> /var/log/inotify.log
    done
fi

Script on vm2 (sion. SH)

#/bin/sh
UNISON=`ps -ef |grep -v grep|grep -c inotifywait`
if [ ${UNISON} -lt 1 ]
then
    other_server="192.168.1.111"
    src1="/home/nfs/"
    dst2="/home/nfs/"
    /usr/bin/inotifywait -mrq -e create,delete,delete_self,modify,move $src1 | while read line; do
        /usr/bin/unison $src1 ssh://$other_server/$dst2
        echo -n "$line " >> /var/log/inotify.log
        echo `date | cut -d " " -f1-4` >> /var/log/inotify.log
    done
fi

Then execute two scripts to achieve two-way synchronization.

5 regularly check the synchronization task (to prevent the process from dying)

[unison@localhost ~]# crontab -e
# vm1
* * * * * nohup unsion.sh > /dev/null 2>&1 &

# vm2
* * * * * nohup unsion.sh > /dev/null 2>&1 &

Keywords: Linux

Added by dgiberson on Wed, 19 Jan 2022 11:55:31 +0200