colony
- Organize many machines together and provide services as a whole
- Clusters can be flexible in terms of scalability and performance
- Cluster classification:
- Load Balance cluster: Load Balance
- High Availability cluster: High Availability
- High Performance Computing
LVS
-
LVS: Linux Virtual Server
-
Load balancing cluster
-
Author: Zhang wensong. Prepared by University of national defense science and technology during doctoral study
-
Working mode of LVS:
- NAT: network address translation
- DR: routing mode
- TUN: tunnel mode
-
Terminology:
- Scheduler: LVS server
- Real Server: the server that provides services
- VIP: virtual address, the address provided for users to access
- DIP: specify the address, which is the address on the LVS server to communicate with the real server
- RIP: real address, the address of the real server
-
There are 10 common scheduling algorithms, including 4 commonly used ones:
- Polling rr: Real Server provides services in turn
- Weighted polling wrr: set the weight according to the server performance, and the request with heavy weight will get more requests
- Minimum connection lc: allocate the request according to the number of connections of the Real Server
- Weighted least connected wlc: similar to wrr, the request is allocated according to the weight
Configure LVS NAT mode
[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-yZOvykLN-1644236949340)(... / imgs/image-20211102101942994.png)]
- Environmental preparation
- client1: eth0 - > 192.168.4.10, gateway 192.168.4.5
- lvs1: eth0 -> 192.168.4.5；eth1->192.168.2.5
- web1: eth1->192.168.2.100； Gateway 192.168.2.5
- web2: eth1->192.168.2.200； Gateway 192.168.2.5
# Create 4 virtual machines
[root@zzgrhel8 ~]# clone-vm7
Enter VM number: 1 # The number filled in here is the virtual machine number
[root@zzgrhel8 ~]# clone-vm7
Enter VM number: 2
[root@zzgrhel8 ~]# clone-vm7
Enter VM number: 3
[root@zzgrhel8 ~]# clone-vm7
Enter VM number: 4
# View virtual machines
[root@zzgrhel8 ~]# virsh list --all
Id name state
--------------------------
- tedu_node01 close
- tedu_node02 close
- tedu_node03 close
- tedu_node04 close
# Start the virtual machine
[root@zzgrhel8 ~]# for i in {1..4}
> do
> virsh start tedu_node0$i
> done
# Initialize virtual machine
[root@zzgrhel8 ~]# virsh console tedu_node01 # Connect tedu_node01 console
localhost login: root
Password: 123456
# After logging in, paste the following contents into the terminal
hostnamectl set-hostname client1
nmcli connection modify eth0 ipv4.method manual ipv4.addresses 192.168.4.10/24
nmcli connection down eth0
nmcli connection up eth0
echo a | passwd --stdin root
nmcli connection modify eth0 ipv4.gateway 192.168.4.5
nmcli connection down eth0
nmcli connection up eth0
# sign out
[root@localhost ~]# exit
# After exiting, press ctrl +] to return to the real machine
# The real machine connects client1 through ssh
[root@zzgrhel8 ~]# rm -f ~/.ssh/known_hosts
[root@zzgrhel8 ~]# ssh 192.168.4.10
# Configure the second machine as lvs1
[root@zzgrhel8 ~]# virsh console tedu_node02
Kernel 3.10.0-862.el7.x86_64 on an x86_64
localhost login: root
Password: 123456
# After logging in, copy the following to the command line
hostnamectl set-hostname lvs1
nmcli connection modify eth0 ipv4.method manual ipv4.addresses 192.168.4.5/24
nmcli connection down eth0
nmcli connection up eth0
nmcli connection modify eth1 ipv4.method manual ipv4.addresses 192.168.2.5/24
nmcli connection down eth1
nmcli connection up eth1
echo a | passwd --stdin root
# sign out
[root@localhost ~]# exit
# After exiting, press ctrl +] to return to the real machine
# The real machine connects to lvs1 through ssh
[root@zzgrhel8 ~]# ssh 192.168.4.5
# Configure the third machine as web1
[root@zzgrhel8 ~]# virsh console tedu_node03
localhost login: root
Password: 123456
# After logging in, copy the following to the command line
hostnamectl set-hostname web1
nmcli connection modify eth1 ipv4.method manual ipv4.addresses 192.168.2.100/24
nmcli connection down eth1
nmcli connection up eth1
nmcli connection modify eth1 ipv4.gateway 192.168.2.5
nmcli connection down eth1
nmcli connection up eth1
echo a | passwd --stdin root
# sign out
[root@localhost ~]# exit
# After exiting, press ctrl +] to return to the real machine
# The real machine connects to web1 via ssh
[root@zzgrhel8 ~]# ssh 192.168.2.100
# Configure the 4th machine as web2
[root@zzgrhel8 ~]# virsh console tedu_node04
localhost login: root
Password: 123456
# After logging in, copy the following to the command line
hostnamectl set-hostname web2
nmcli connection modify eth1 ipv4.method manual ipv4.addresses 192.168.2.200/24
nmcli connection down eth1
nmcli connection up eth1
nmcli connection modify eth1 ipv4.gateway 192.168.2.5
nmcli connection down eth1
nmcli connection up eth1
echo a | passwd --stdin root
# sign out
[root@localhost ~]# exit
# After exiting, press ctrl +] to return to the real machine
# The real machine connects to web2 through ssh
[root@zzgrhel8 ~]# ssh 192.168.2.200
- If the virtual machine created through clone-vm7 has the address of 192.168.4.0 network segment, yum has been configured. The virtual machine has turned off selinux and firewall.
To configure LVS NAT mode
- Configure 2 web servers
[root@web1 ~]# vim /etc/yum.repos.d/local.repo [local_repo] name=CentOS-$releasever - Base baseurl=ftp://192.168.2.254/centos-1804 enabled=1 gpgcheck=0 [root@web2 ~]# vim /etc/yum.repos.d/local.repo [local_repo] name=CentOS-$releasever - Base baseurl=ftp://192.168.2.254/centos-1804 enabled=1 gpgcheck=0 [root@web1 ~]# yum install -y httpd [root@web2 ~]# yum install -y httpd # Create test page [root@web1 ~]# echo "192.168.2.100" > /var/www/html/index.html [root@web2 ~]# echo "apache web server2" > /var/www/html/index.html [root@web1 ~]# systemctl enable httpd --now [root@web2 ~]# systemctl enable httpd --now # Test access to web server on lvs1 [root@lvs1 ~]# curl http://192.168.2.100/ 192.168.2.100 [root@lvs1 ~]# ^100^200 # Replace 100 in the previous command with 200 and execute curl http://192.168.2.200/ apache web server2
- Make sure that the ip forwarding function of lvs1 is turned on. This function requires changing kernel parameters
# View the kernel parameters of ip forwarding function [root@lvs1 ~]# sysctl -a # View all kernel parameters [root@lvs1 ~]# sysctl -a | grep ip_forward # View ip_foward parameter net.ipv4.ip_forward = 1 # 1 means to turn on Forwarding and 0 means to turn off forwarding # Permanently set on ip_forward function [root@lvs1 ~]# echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf [root@lvs1 ~]# sysctl -p # Test access from client to server [root@client1 ~]# curl http://192.168.2.100/ 192.168.2.100 [root@client1 ~]# curl http://192.168.2.200/ apache web server2
- Install LVS
[root@lvs1 ~]# yum install -y ipvsadm
- ipvsadm instructions
[root@lvs1 ~]# ipvsadm -A: Add virtual server -E: Edit virtual server -D: Delete virtual server -t: Add real tcp The server -u: Add real udp The server -s: Specify the scheduling algorithm. Such as polling rr/Weighted polling wrr/Minimum connection lc/Weighted least connection wlc -a: After adding a virtual server, add a real server to the virtual server -r: Specify the real server -w: Set weight -m: The specified working mode is NAT -g: The specified working mode is DR
- Configure LVS
# Create a virtual server for the web server and use the rr scheduling algorithm
[root@lvs1 ~]# ipvsadm -A -t 192.168.4.5:80 -s rr
# View configuration
[root@lvs1 ~]# ipvsadm -Ln
# Add RIP to virtual server
[root@lvs1 ~]# ipvsadm -a -t 192.168.4.5:80 -r 192.168.2.100 -w 1 -m
[root@lvs1 ~]# ipvsadm -a -t 192.168.4.5:80 -r 192.168.2.200 -w 2 -m
# View configuration
[root@lvs1 ~]# ipvsadm -Ln
# verification
[root@client1 ~]# for i in {1..4}
> do
> curl http://192.168.4.5/
> done
apache web server2
192.168.2.100
apache web server2
192.168.2.100
# Delete configuration. (if the configuration is wrong, use the following command to delete the reconfiguration)
[root@lvs1 ~]# ipvsadm -D -t 192.168.4.5:80
# Modify the scheduling mode to weighted polling
[root@lvs1 ~]# ipvsadm -E -t 192.168.4.5:80 -s wrr
# Verify configuration
[root@client1 ~]# for i in {1..6}; do curl http://192.168.4.5/; done
apache web server2
apache web server2
192.168.2.100
apache web server2
apache web server2
192.168.2.100
LVS DR mode
[the external link image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-TceI136X-1644236949342)(... / imgs/image-20211102151338519.png)]
- In LVS DR mode, both LVS host and web server are single network cards. They are connected in the same network
- Modify the experimental environment
- client1: eth0-> 192.168.4.10
- lvs1: eth0 - > 192.168.4.5, delete the IP address of eth1
- web1: eth0 - > 192.168.4.100, delete the IP address of eth1
- web2: eth0 - > 192.168.4.200, delete the IP address of eth1
# Delete lvs virtual server configuration [root@lvs1 ~]# ipvsadm -D -t 192.168.4.5:80 [root@lvs1 ~]# ipvsadm -Ln # Delete the configuration of eth1 on lvs1 [root@lvs1 ~]# nmcli connection modify eth1 ipv4.method disabled ipv4.addresses '' [root@lvs1 ~]# ifdown eth1 # Modify the configuration of web1: stop the address of eth1. The address of eth0 is 192.168.4.100 # Enter the network card configuration file directory [root@web1 ~]# cd /etc/sysconfig/network-scripts/ # The configuration file of eth0 network card is ifcfg-eth0 [root@web1 network-scripts]# ls ifcfg-eth* ifcfg-eth0 ifcfg-eth1 ifcfg-eth2 ifcfg-eth3 # Configure eth0 address [root@web1 network-scripts]# vim ifcfg-eth0 TYPE=Ethernet # The network type is Ethernet BOOTPROTO=none # The IP address is statically configured, or static can be used NAME=eth0 # Rename device DEVICE=eth0 # Network card device name ONBOOT=yes # Boot and activate network card IPADDR=192.168.4.100 # IP address NETMASK=255.255.255.0 # Subnet mask GATEWAY=192.168.4.254 # gateway [root@web1 network-scripts]# ifdown eth0; ifup eth0 # Disable active network card # Stop eth1 on web1 [root@web1 ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth1 TYPE=Ethernet BOOTPROTO=none NAME=eth1 DEVICE=eth1 ONBOOT=no [root@web1 ~]# ifdown eth1 # Modify the network of web2 [root@web2 ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth0 TYPE=Ethernet BOOTPROTO=none NAME=eth0 DEVICE=eth0 ONBOOT=yes IPADDR=192.168.4.200 NETMASK=255.255.255.0 GATEWAY=192.168.4.254 [root@web2 ~]# ifdown eth0; ifup eth0 [root@web2 ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth1 TYPE=Ethernet BOOTPROTO=none NAME=eth1 DEVICE=eth1 ONBOOT=no [root@web2 ~]# ifdown eth1 # The terminal is stuck, turn it off and reconnect at the new terminal
Configure LVS DR mode
- Configure vip 192.168.4.15 on eth0 of lvs1. Configure vip by creating a logical port for eth0, and name the logical port eth0:0
[root@lvs1 ~]# cd /etc/sysconfig/network-scripts/ [root@lvs1 network-scripts]# cp ifcfg-eth0 ifcfg-eth0:0 [root@lvs1 network-scripts]# vim ifcfg-eth0:0 TYPE=Ethernet BOOTPROTO=none NAME=eth0:0 DEVICE=eth0:0 ONBOOT=yes IPADDR=192.168.4.15 PREFIX=24 [root@lvs1 network-scripts]# ifup eth0:0 # View new address [root@lvs1 network-scripts]# ifconfig
- Configure vip 192.168.4.15 on the lo of two web servers
[root@web1 ~]# cd /etc/sysconfig/network-scripts/ [root@web1 network-scripts]# cp ifcfg-lo ifcfg-lo:0 [root@web1 network-scripts]# vim ifcfg-lo:0 DEVICE=lo:0 IPADDR=192.168.4.15 NETMASK=255.255.255.255 NETWORK=192.168.4.15 BROADCAST=192.168.4.15 ONBOOT=yes NAME=lo:0 [root@web1 network-scripts]# ifup lo:0 [root@web1 network-scripts]# ifconfig # Copy the configuration of web1 to web2 [root@web1 network-scripts]# scp ./ifcfg-lo:0 192.168.4.200:/etc/sysconfig/network-scripts/ [root@web2 ~]# ifup lo:0 [root@web2 ~]# ifconfig
- Configure the kernel parameters on the two web servers so that they do not respond to the request of 192.168.4.15
[root@web1 ~]# sysctl -a | grep arp_ignore net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.lo.arp_ignore = 0 [root@web1 ~]# sysctl -a | grep arp_announce net.ipv4.conf.all.arp_announce = 2 net.ipv4.conf.lo.arp_announce = 0 [root@web1 ~]# vim /etc/sysctl.conf net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.lo.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 net.ipv4.conf.lo.arp_announce = 2 [root@web1 ~]# sysctl -p [root@web2 ~]# vim /etc/sysctl.conf net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.lo.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 net.ipv4.conf.lo.arp_announce = 2 [root@web2 ~]# sysctl -p
- Configuring a virtual server on lvs1
# Create virtual server
[root@lvs1 ~]# ipvsadm -A -t 192.168.4.15:80 -s wlc
# Adding a real server to a virtual server
[root@lvs1 ~]# ipvsadm -a -t 192.168.4.15:80 -r 192.168.4.100 -w 1 -g
[root@lvs1 ~]# ipvsadm -a -t 192.168.4.15:80 -r 192.168.4.200 -w 2 -g
# View configuration
[root@lvs1 ~]# ipvsadm -Ln
# Customer verification
[root@client1 ~]# for i in {1..6}; do curl http://192.168.4.15/; done
apache web server2
192.168.2.100
apache web server2
apache web server2
192.168.2.100
apache web server2
- lvs itself has no application server monitoring function. If there is a problem with the web server, it needs to be deleted from the rules manually
[root@web1 ~]# systemctl stop httpd
# When the client accesses, the request forwarded to web1 will reject the connection
[root@client1 ~]# for i in {1..6}; do curl http://192.168.4.15/; done
apache web server2
curl: (7) Failed connect to 192.168.4.15:80; connection denied
apache web server2
apache web server2
curl: (7) Failed connect to 192.168.4.15:80; connection denied
apache web server2
# Delete web1 from rules
[root@lvs1 ~]# ipvsadm -d -t 192.168.4.15:80 -r 192.168.4.100
[root@client1 ~]# for i in {1..6}; do curl http://192.168.4.15/; done
apache web server2
apache web server2
apache web server2
apache web server2
apache web server2
apache web server2
Attachment: troubleshooting steps in case of error:
# The web server can be accessed on lvs
[root@lvs1 ~]# curl http://192.168.4.100/
192.168.2.100
[root@lvs1 ~]# curl http://192.168.4.200/
apache web server2
# View vip
[root@lvs1 ~]# ifconfig eth0:0
eth0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.4.15 netmask 255.255.255.0 broadcast 192.168.4.255
ether 52:54:00:0d:fb:79 txqueuelen 1000 (Ethernet)
[root@web1 ~]# ifconfig lo:0
lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 192.168.4.15 netmask 255.255.255.255
loop txqueuelen 1000 (Local Loopback)
# View kernel parameters
[root@web1 ~]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
# View rules
[root@lvs1 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.4.15:80 wlc
-> 192.168.4.100:80 Route 1 0 0
-> 192.168.4.200:80 Route 2 0 0