Installation and use of rsync, gitlab, svn
1. Introduction to Rsync
rsync is a data mirror backup tool for linux systems. Remote Sync, a fast incremental backup tool, allows remote synchronization, supports local replication, or synchronization with other SSH, rsync hosts.
2.rsync characteristics
rsync supports many features:
You can mirror the entire directory tree and file system
It is easy to keep the permissions, time, hard and soft links of the original file, etc.
Install without special privileges
Fast: rsync copies everything the first time it synchronizes, but transfers only the modified files the next time. rsync can compress and decompress data during transmission, so it can use less bandwidth
Security: You can use scp, ssh, etc. to transfer files or, of course, direct socket connections
Support anonymous transfer for easy site mirroring
3.rsync's ssh authentication protocol
The rsync command must log on to the remote host authentication before synchronizing system files. There are two protocols used in the authentication process:
ssh protocol
rsync protocol
rsync server`End does not start`rsync`Of`daemon`Process, as long as get`remote host`The username and password are straightforward`rsync`synchronize files `rsync server`End because no startup is required`daemon`Process, so no profile is needed`/etc/rsyncd.conf
The SSH authentication protocol works the same way as scp. If you do not want to enter a password during synchronization, use ssh-keygen-t RSA to get through the channel
//This defaults to omitting -e ssh, which is equivalent to the following:
rsync -avz /SRC -e ssh root@172.16.12.129:/DEST
-a //File host change, timestamp unchanged
-z //Compressed Data Transfer
//When we encounter a port to be modified, we can:
rsync -avz /SRC -e "ssh -p2222" root@172.16.12.129:/DEST
//Modified the port of ssh protocol, default is 22
4.rsync command
//Rsync has three common command formats:
rsync [OPTION]... SRC DEST
rsync [OPTION]... SRC [USER@]HOST:DEST
rsync [OPTION]... [USER@]HOST:SRC DEST
//rsync has three different modes of operation corresponding to the above three command formats:
1)Copy the local file. SRC and DES Path information does not contain a single colon":"This mode of operation is initiated when the separator is used. For example:
[root@localhost ~]# ls
anaconda-ks.cfg nfs.sh
[root@localhost ~]# rsync -a nfs.sh a.sh
[root@localhost ~]# ll
total 12
-rw-------. 1 root root 1453 Jun 13 19:27 anaconda-ks.cfg
-rwxr-xr-x 1 root root 1041 Aug 8 18:14 a.sh
-rwxr-xr-x 1 root root 1041 Aug 8 18:14 nfs.sh
[root@localhost ~]# ll -i
total 12
33574978 -rw-------. 1 root root 1453 Jun 13 19:27 anaconda-ks.cfg
33574979 -rwxr-xr-x 1 root root 1041 Aug 8 18:14 a.sh
33574990 -rwxr-xr-x 1 root root 1041 Aug 8 18:14 nfs.sh
2)Use a remote shell program(as rsh,ssh)Implements copying the contents of a local machine to a remote machine. DST Path Address Pack \
Contains a single colon":"Start the mode when the separator is used. For example:
[root@localhost ~]# rsync -avz nfs.sh root@172.16.12.129:/root/b.sh
sending incremental file list
nfs.sh
sent 643 bytes received 35 bytes 1,356.00 bytes/sec
total size is 1,041 speedup is 1.54
[root@localhost ~]# ssh root@172.16.12.129 'ls -l /root'
total 8
-rw-------. 1 root root 1454 Aug 6 04:39 anaconda-ks.cfg
-rwxr-xr-x 1 root root 1041 Aug 8 2018 b.sh
3)Use a remote shell program(as rsh,ssh)Implements copying the contents of a remote machine to a local machine. SRC Address Path \
Contains a single colon":"Start the mode when the separator is used. For example:
[root@localhost ~]# ls
anaconda-ks.cfg a.sh nfs.sh
[root@localhost ~]# rsync -avz root@172.16.12.129:/etc/yum.repos.d /root/
receiving incremental file list
yum.repos.d/
yum.repos.d/163.repo
yum.repos.d/redhat.repo
sent 66 bytes received 918 bytes 1,968.00 bytes/sec
total size is 1,820 speedup is 1.85
[root@localhost ~]# ls
anaconda-ks.cfg a.sh nfs.sh yum.repos.d
[root@localhost ~]# ls yum.repos.d/
163.repo redhat.repo
//Common rsync options:
-a, --archive //File
-v, --verbose //Verbose Mode
-q, --quiet //silent mode
-r, --recursive //recursion
-p, --perms //Keep original permission properties
-z, --compress //Compress during transmission, save bandwidth and speed up transmission
--delete //Deletions made on the source server will also be synchronized on the target server
5.rsync+inotify
rsync Traditional cp,tar Compared to backup methods, rsync It has the advantages of high security, fast backup and incremental backup support. rsync It can solve the data backup requirements that are not very real-time, such as periodic backup of file server data to remote servers, periodic data mirroring of local disks, etc. As the scale of the application system expands, there are better requirements for data security and reliability. rsync In high-end business systems, there are also a number of shortcomings gradually exposed, first of all, rsync When synchronizing data, all files need to be scanned, compared, and differentially transferred. If the number of files reaches millions or even millions of magnitudes, scanning all files can be time consuming. And it is often a very inefficient way to do so because very few of them are changing. rsync Can't monitor and synchronize data in real time, although it can pass linux The daemon triggers synchronization in such a way that there must be a time difference between the two triggers, which may result in inconsistencies between the server and client data that cannot be fully recovered in the event of application failure. rsync+inotify A combination appears! Inotify It is a powerful, fine-grained, asynchronous file system event monitoring mechanism. linux Kernel from 2.6.13 Beginning, joined Inotify Support, through Inotify With this kernel interface, third-party software can monitor the changes of files in the file system. inotify-tools It is such a third-party software. As mentioned earlier, rsync Triggered file synchronization is possible, but via crontab The daemon triggers in such a way that the synchronized data differs from the actual data. inotify It monitors changes in the file system and triggers when there are any changes to the file rsync Synchronization, which just solves the real-time problem of synchronizing data.
Synchronize the source server/etc directory in real time to the target server/tmp/
Do something on the target server
//Close firewall and selinux [root@localhost ~]# systemctl stop firewalld [root@localhost ~]# systemctl disable firewalld Removed /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. [root@localhost ~]# getenforce Enforcing [root@localhost ~]# setenforce 0 [root@localhost ~]# sed -ri 's/^(SELINUX=).*/\1disabled/g' /etc/sysconfig/selinux //Install rsync server software [root@localhost ~]# yum -y install rsync rsync-daemon Last Metadata Expiration Check:23:10:53 Previously, it was executed on Sunday, October 10, 2021 at 23.32 minutes and 12 seconds. software package rsync-3.1.3-11.el8.x86_64 Installed. Dependency resolution. Set up rsyncd.conf configuration file [root@localhost ~]# cat /etc/rsyncd.conf log file = /var/log/rsyncd.log pidfile = /var/run/rsyncd.pid lock file = /var/run/rsync.lock secrets file = /etc/rsync.pass [etc_from_client] path = /tmp/ comment = sync etc from client uid = root gid = root port = 873 ignore errors read only = no list = no max connections = 200 timeout = 600 auth users = admin hosts allow = 192.168.197.131 //Create user authentication file [root@localhost ~]# echo 'admin:123456' > /etc/rsync.pass [root@localhost ~]# cat /etc/rsync.pass admin:123456 //Set User Permissions [root@localhost ~]# chmod 600 /etc/rsync* [root@localhost ~]# ll /etc/rsync* -rw-------. 1 root root 885 10 January 1122:52 /etc/rsyncd.conf -rw-------. 1 root root 13 10 January 1122:54 /etc/rsync.pass //Start the rsync service and set up boot-up self-start [root@localhost ~]# systemctl start rsyncd [root@localhost ~]# systemctl enable rsyncd Created symlink /etc/systemd/system/multi-user.target.wants/rsyncd.service → /usr/lib/systemd/system/rsyncd.service. [root@localhost ~]# ss -antl State Recv-Q Send-Q Local Address:Port Peer Address:Port Process LISTEN 0 5 0.0.0.0:873 0.0.0.0:* LISTEN 0 128 0.0.0.0:111 0.0.0.0:* LISTEN 0 32 192.168.122.1:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 5 127.0.0.1:631 0.0.0.0:* LISTEN 0 5 [::]:873 [::]:* LISTEN 0 80 *:3306 *:* LISTEN 0 128 [::]:111 [::]:* LISTEN 0 128 [::]:22 [::]:* LISTEN 0 5 [::1]:631 [::]:*
Do the following on the source server:
//Close firewall and selinux
[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]# systemctl disable firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@localhost ~]# getenforce 0
Enforcing
[root@localhost ~]# setenforce 0
[root@localhost ~]# sed -ri 's/^(SELINUX=).*/\1disabled/g' /etc/sysconfig/selinux
[root@localhost ~]# cd /etc/yum.repos.d/
[root@localhost yum.repos.d]# wget http://mirrors.163.com/.help/CentOS7-Base-163.repo
--2021-10-11 23:02:41-- http://mirrors.163.com/.help/CentOS7-Base-163.repo
Resolving Host mirrors.163.com (mirrors.163.com)... 59.111.0.251
on connection mirrors.163.com (mirrors.163.com)|59.111.0.251|:80... Connected.
Sent HTTP Request, awaiting response... 200 OK
Length: 1572 (1.5K) [application/octet-stream]
Saving to: "CentOS7-Base-163.repo"
CentOS7-Base-163.repo 100%[======================>] 1.54K --.-KB/s Time-consuming 0 s
2021-10-11 23:02:41 (56.7 MB/s) - Saved" CentOS7-Base-163.repo" [1572/1572])
[root@localhost yum.repos.d]# cd
[root@localhost ~]# sed -i 's/\$releasever/7/g' /etc/yum.repos.d/CentOS7-Base-163.repo
[root@localhost ~]# sed -i 's/^enabled=.*/enabled=1/g' /etc/yum.repos.d/CentOS7-Base-163.repo
[root@localhost ~]# yum -y install epel-release
[root@localhost ~]# yum -y install rsync
Extra Packages for Enterprise Linux 7 - x86_64 2.5 MB/s | 17 MB 00:06
Last Metadata Expiration Check:0:00:04 Previously, it was executed on Monday, 11 October 2021 at 23.04 minutes and 34 seconds.
software package rsync-3.1.3-11.el8.x86_64 Installed.
Dependency resolution.
No processing is required.
//Create Authentication Password File
[root@localhost ~]# echo '123456' > /etc/rsync.pass
[root@localhost ~]# cat /etc/rsync.pass
123456
[root@localhost ~]# chmod 600 /etc/rsync.pass
//Create Test Directory
[root@localhost ~]# mkdir -pv /root/etc/test
mkdir: Directory created '/root/etc'
mkdir: Directory created '/root/etc/test'
[root@localhost ~]# rsync -avH --port 873 --progress --delete /root/etc/ admin@192.168.197.128::etc_from_client --password-file=/etc/rsync.pass
sending incremental file list
deleting vmware-root_924-2722763428/
deleting vmware-root_903-3979774182/
deleting tracker-extract-files.1000/
deleting tracker-extract-files.0/
deleting systemd-private-ab9f8f7ff4744405af6888ae91fe702b-rtkit-daemon.service-g7Bco8/tmp/
deleting systemd-private-ab9f8f7ff4744405af6888ae91fe702b-rtkit-daemon.service-g7Bco8/
deleting systemd-private-ab9f8f7ff4744405af6888ae91fe702b-mariadb.service-kKwmzS/tmp/
deleting systemd-private-ab9f8f7ff4744405af6888ae91fe702b-mariadb.service-kKwmzS/
deleting systemd-private-ab9f8f7ff4744405af6888ae91fe702b-geoclue.service-drC8Zv/tmp/
deleting systemd-private-ab9f8f7ff4744405af6888ae91fe702b-geoclue.service-drC8Zv/
deleting systemd-private-ab9f8f7ff4744405af6888ae91fe702b-fwupd.service-leCjdT/tmp/
deleting systemd-private-ab9f8f7ff4744405af6888ae91fe702b-fwupd.service-leCjdT/
deleting systemd-private-ab9f8f7ff4744405af6888ae91fe702b-colord.service-zoT1xA/tmp/
deleting systemd-private-ab9f8f7ff4744405af6888ae91fe702b-colord.service-zoT1xA/
deleting systemd-private-ab9f8f7ff4744405af6888ae91fe702b-chronyd.service-2OVAlF/tmp/
deleting systemd-private-ab9f8f7ff4744405af6888ae91fe702b-chronyd.service-2OVAlF/
deleting systemd-private-ab9f8f7ff4744405af6888ae91fe702b-ModemManager.service-AWAetc/tmp/
deleting systemd-private-ab9f8f7ff4744405af6888ae91fe702b-ModemManager.service-AWAetc/
deleting .local/share/ibus-typing-booster/data/
deleting .local/share/ibus-typing-booster/
deleting .local/share/
deleting .local/
deleting .font-unix/
deleting .esd-975/
deleting .esd-1000/socket
deleting .esd-1000/
deleting .esd-0/socket
deleting .esd-0/
deleting .XIM-unix/
deleting .X11-unix/X1024
deleting .X11-unix/X0
deleting .X11-unix/
deleting .Test-unix/
deleting .ICE-unix/2153
deleting .ICE-unix/1492
deleting .ICE-unix/
deleting sensitive-info.log
deleting program.log
deleting packaging.log
deleting ks-script-sjvqx3ij
deleting dbus.log
deleting anaconda.log
deleting .viminfo
deleting .X1024-lock
deleting .X0-lock
./
test/
sent 77 bytes received 1,722 bytes 719.60 bytes/sec
total size is 0 speedup is 0.00
//Install the inotify-tools tool to trigger rsync synchronization in real time
[root@localhost ~]# yum -y install make gcc gcc-c++ inotify-tools
Last Metadata Expiration Check:0:05:14 Previously, it was executed on Monday, 11 October 2021 at 23.04 minutes and 34 seconds.
software package make-1:4.2.1-10.el8.x86_64 Installed.
software package gcc-8.4.1-1.el8.x86_64 Installed.
software package gcc-c++-8.4.1-1.el8.x86_64 Installed.
Dependency resolution.
[root@localhost ~]# mkdir /scripts
[root@localhost ~]# touch /scripts/inotify.sh
[root@localhost ~]# chmod 755 /scripts/inotify.sh
[root@localhost ~]# cat /scripts/inotify.sh
host=192.168.197.128
src=/etc
des=etc_from_client
password=/etc/rsync.pass
inotifywait=/usr/bin/inotifywait
$inotifywait -mrq --timefmt '%Y%m%d %H:%M' --format '%T %w%f%e' -e modify,delete,create,attrib $src \
| while read files;do
rsync -avzP --delete --timeout=100 --password-file=${password} $src $user@$host::$des
echo "${files} was rsynced" >>/tmp/rsync.log 2>&1
done
//start-up
[root@localhost ~]# nohup bash /scripts/inotify.sh >/dev/null 2>&1&
[5] 1729334
[4] Completed nohup bash /scripts/inotify.sh > /dev/null 2>&1
//Generate a new file on the source server
[root@localhost ~]# ls /etc/yum.repos.d/
CentOS-Stream-AppStream.repo
CentOS-Stream-BaseOS.repo
[root@localhost ~]# echo 'hello world' > /etc/yum.repos.d/test
//View inotify generated logs
[root@localhost ~]# tail /tmp/rsync.log
//Set script to start automatically
[root@localhost ~]# chmod +x /etc/rc.d/rc.local
[root@localhost ~]# ll /etc/rc.d/rc.local
-rwxr-xr-x 1 root root 473 Aug 10 23:23 /etc/rc.d/rc.local
[root@localhost ~]# echo 'nohup /bin/bash /scripts/inotify.sh' >> /etc/rc.d/rc.local
[root@localhost ~]# tail /etc/rc.d/rc.local
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.
touch /var/lock/subsys/local
nohup /bin/bash /scripts/inotify.sh
[root@localhost tmp]# ls /etc/yum.repos.d
CentOS-Stream-AppStream.repo
CentOS-Stream-BaseOS.repo
gitlab
1. Introduction to version control
Version control refers to the management of changes in various program codes, configuration files and description documents in the software development process. It is one of the core ideas of software configuration management.
The most important function of version control is to track changes to a file. It faithfully records when and who changed what was in the file. Each time a file changes, the version number of the file increases. In addition to recording version changes, another important function of version control is parallel development. Software development is often done in collaboration with multiple people, with versions.Control can effectively solve the synchronization of versions and development communication problems between different developers, and improve the efficiency of collaborative development. Bug correction problems of the most common different versions of software in parallel development can also be effectively solved by branch and merge in version control.
Specifically, in each development task, a development baseline is set to determine the initial version of each configuration item. During the development process, the developer develops the required target version based on the version of the development baseline. When a requirement change occurs, the scope of the change is determined by evaluating the change, and the version of the affected configuration item is repaired.Change so that the version tree of a configuration item continues to extend or branch to form a new target version, depending on the nature of the change. No changes should be made to configuration items that are unaffected by the change. At the same time, the impact of the change on the version should be recorded and tracked. You can also revert to previous versions if necessary. For example, when development requirements or requirements changes are takenThe ability to roll back versions to the development baseline version is required at the time of elimination. In the quarterly upgrade package unpacking and reorganization process that has occurred, it is essentially to roll back versions of some configuration items to the development baseline, recombine different branches corresponding to different requirements, and form new upgrade package versions.
Version control is the core function of software configuration management. All elements placed in the configuration library should automatically identify the version and ensure the uniqueness of the version naming. Versions are automatically branched and evolved according to the set usage model during the generation process. In addition to the version information recorded automatically by the system, it needs to be determined in order to fit the various stages of the software development process.Defines, collects some metadata to record versions of ancillary information, standardizes development processes, and prepares for future measurements of software processes. Of course, if the tools are selected, these ancillary data will directly count out process data, which will facilitate software process improvement activities. For each baseline control item in the configuration library, it should be based on its baseline location andStatus sets the appropriate access rights. Generally speaking, versions prior to the baseline version should be locked, and changes to them should be made according to the change control process.
Common version control tools:
gitlab subversion
2.gitlab deployment
[root@localhost ~]# yum -y install epel-release Last Metadata Expiration Check:0:23:24 Previously, it was executed on Monday, 11 October 2021 at 23.04 minutes and 34 seconds. software package epel-release-7-11.noarch Installed. Dependency resolution. ===================================================================================== software package Framework Edition Warehouse Size ===================================================================================== upgrade: [root@localhost ~]# yum -y install git curl openssh-server openssh-clients postfix cronie policycoreutils-python-utils Extra Packages for Enterprise Linux 7 - x86_64 3.7 MB/s | 17 MB 00:04 Last Metadata Expiration Check:0:00:02 Previously, it was executed on Monday, 11 October 2021, at 23.28 minutes and 31 seconds. software package curl-7.61.1-17.el8.x86_64 Installed. software package openssh-server-8.0p1-5.el8.x86_64 Installed. [root@localhost ~]# systemctl restart postfix [root@localhost ~]# systemctl enable postfix Created symlink /etc/systemd/system/multi-user.target.wants/postfix.service → /usr/lib/systemd/system/postfix.service. [root@localhost ~]# wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el8/gitlab-ce-14.3.2-ce.0.el8.x86_64.rpm Saving to: "gitlab-ce-14.3.2-ce.0.el8.x86_64.rpm" gitlab-ce-14.3.2-ce.0 100%[======================>] 917.02M 10.7MB/s Time-consuming 90 s 2021-10-11 23:31:44 (10.2 MB/s) - Saved" gitlab-ce-14.3.2-ce.0.el8.x86_64.rpm" [961561901/961561901]) [root@localhost ~]# rpm -ivh gitlab-ce-14.3.2-ce.0.el8.x86_64.rpm Warning: gitlab-ce-14.3.2-ce.0.el8.x86_64.rpm: head V4 RSA/SHA256 Signature, secret key ID f27eab47: NOKEY Verifying... ################################# [100%] Preparing... ################################# [100%] Upgrading/install... 1:gitlab-ce-14.3.2-ce.0.el8 ############################# ( 87%) [root@localhost ~]# vim /etc/gitlab/gitlab.rb external_url 'http://192.168.197.131' //restart [root@localhost ~]# gitlab-ctl reconfigure+
# svn Installation
1. Download
Download address:https://tortoisesvn.net/downloads.html
Enter Interface Installation
