Introduction to cluster and LVS, LVS-NAT cluster, LVS-DR cluster

colony

  • Organize many machines together and provide services as a whole
  • Clusters can be flexible in terms of scalability and performance
  • Cluster classification:
    • Load Balance cluster: Load Balance
    • High Availability cluster: High Availability
    • High Performance Computing

LVS

  • LVS: Linux Virtual Server

  • Load balancing cluster

  • Author: Zhang wensong. Prepared by University of national defense science and technology during doctoral study

  • Working mode of LVS:

    • NAT: network address translation
    • DR: routing mode
    • TUN: tunnel mode
  • Terminology:

    • Scheduler: LVS server
    • Real Server: the server that provides services
    • VIP: virtual address, the address provided for users to access
    • DIP: specify the address, which is the address on the LVS server to communicate with the real server
    • RIP: real address, the address of the real server
  • There are 10 common scheduling algorithms, including 4 commonly used ones:

    • Polling rr: Real Server provides services in turn
    • Weighted polling wrr: set the weight according to the server performance, and the request with heavy weight will get more requests
    • Minimum connection lc: allocate the request according to the number of connections of the Real Server
    • Weighted least connected wlc: similar to wrr, the request is allocated according to the weight

Configure LVS NAT mode

[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-yZOvykLN-1644236949340)(... / imgs/image-20211102101942994.png)]

  • Environmental preparation
    • client1: eth0 - > 192.168.4.10, gateway 192.168.4.5
    • lvs1: eth0 -> 192.168.4.5;eth1->192.168.2.5
    • web1: eth1->192.168.2.100; Gateway 192.168.2.5
    • web2: eth1->192.168.2.200; Gateway 192.168.2.5
# Create 4 virtual machines
[root@zzgrhel8 ~]# clone-vm7 
Enter VM number: 1    # The number filled in here is the virtual machine number
[root@zzgrhel8 ~]# clone-vm7 
Enter VM number: 2
[root@zzgrhel8 ~]# clone-vm7 
Enter VM number: 3
[root@zzgrhel8 ~]# clone-vm7 
Enter VM number: 4

# View virtual machines
[root@zzgrhel8 ~]# virsh list --all
 Id   name          state
--------------------------
 -    tedu_node01   close
 -    tedu_node02   close
 -    tedu_node03   close
 -    tedu_node04   close

# Start the virtual machine
[root@zzgrhel8 ~]# for i in {1..4}
> do
> virsh start tedu_node0$i
> done

# Initialize virtual machine
[root@zzgrhel8 ~]# virsh console tedu_node01  # Connect tedu_node01 console
localhost login: root
Password: 123456
# After logging in, paste the following contents into the terminal
hostnamectl set-hostname client1
nmcli connection modify eth0 ipv4.method manual ipv4.addresses 192.168.4.10/24
nmcli connection down eth0
nmcli connection up eth0
echo a | passwd --stdin root
nmcli connection modify eth0 ipv4.gateway 192.168.4.5
nmcli connection down eth0
nmcli connection up eth0

# sign out
[root@localhost ~]# exit
# After exiting, press ctrl +] to return to the real machine

# The real machine connects client1 through ssh
[root@zzgrhel8 ~]# rm -f ~/.ssh/known_hosts 
[root@zzgrhel8 ~]# ssh 192.168.4.10


# Configure the second machine as lvs1
[root@zzgrhel8 ~]# virsh console tedu_node02
Kernel 3.10.0-862.el7.x86_64 on an x86_64

localhost login: root
Password: 123456

# After logging in, copy the following to the command line
hostnamectl set-hostname lvs1
nmcli connection modify eth0 ipv4.method manual ipv4.addresses 192.168.4.5/24
nmcli connection down eth0
nmcli connection up eth0
nmcli connection modify eth1 ipv4.method manual ipv4.addresses 192.168.2.5/24
nmcli connection down eth1
nmcli connection up eth1
echo a | passwd --stdin root

# sign out
[root@localhost ~]# exit
# After exiting, press ctrl +] to return to the real machine

# The real machine connects to lvs1 through ssh
[root@zzgrhel8 ~]# ssh 192.168.4.5



# Configure the third machine as web1
[root@zzgrhel8 ~]# virsh console tedu_node03
localhost login: root
Password: 123456

# After logging in, copy the following to the command line
hostnamectl set-hostname web1
nmcli connection modify eth1 ipv4.method manual ipv4.addresses 192.168.2.100/24
nmcli connection down eth1
nmcli connection up eth1
nmcli connection modify eth1 ipv4.gateway 192.168.2.5
nmcli connection down eth1
nmcli connection up eth1
echo a | passwd --stdin root

# sign out
[root@localhost ~]# exit
# After exiting, press ctrl +] to return to the real machine

# The real machine connects to web1 via ssh
[root@zzgrhel8 ~]# ssh 192.168.2.100




# Configure the 4th machine as web2
[root@zzgrhel8 ~]# virsh console tedu_node04
localhost login: root
Password: 123456

# After logging in, copy the following to the command line
hostnamectl set-hostname web2
nmcli connection modify eth1 ipv4.method manual ipv4.addresses 192.168.2.200/24
nmcli connection down eth1
nmcli connection up eth1
nmcli connection modify eth1 ipv4.gateway 192.168.2.5
nmcli connection down eth1
nmcli connection up eth1
echo a | passwd --stdin root

# sign out
[root@localhost ~]# exit
# After exiting, press ctrl +] to return to the real machine

# The real machine connects to web2 through ssh
[root@zzgrhel8 ~]# ssh 192.168.2.200
  • If the virtual machine created through clone-vm7 has the address of 192.168.4.0 network segment, yum has been configured. The virtual machine has turned off selinux and firewall.

To configure LVS NAT mode

  • Configure 2 web servers
[root@web1 ~]# vim /etc/yum.repos.d/local.repo 
[local_repo]
name=CentOS-$releasever - Base
baseurl=ftp://192.168.2.254/centos-1804
enabled=1
gpgcheck=0
[root@web2 ~]# vim /etc/yum.repos.d/local.repo 
[local_repo]
name=CentOS-$releasever - Base
baseurl=ftp://192.168.2.254/centos-1804
enabled=1
gpgcheck=0
[root@web1 ~]# yum install -y httpd
[root@web2 ~]# yum install -y httpd

# Create test page
[root@web1 ~]# echo "192.168.2.100" > /var/www/html/index.html
[root@web2 ~]# echo "apache web server2" > /var/www/html/index.html
[root@web1 ~]# systemctl enable httpd --now
[root@web2 ~]# systemctl enable httpd --now

# Test access to web server on lvs1
[root@lvs1 ~]# curl http://192.168.2.100/
192.168.2.100
[root@lvs1 ~]# ^100^200   # Replace 100 in the previous command with 200 and execute
curl http://192.168.2.200/
apache web server2
  • Make sure that the ip forwarding function of lvs1 is turned on. This function requires changing kernel parameters
# View the kernel parameters of ip forwarding function
[root@lvs1 ~]# sysctl -a    # View all kernel parameters
[root@lvs1 ~]# sysctl -a | grep ip_forward  # View ip_foward parameter
net.ipv4.ip_forward = 1   # 1 means to turn on Forwarding and 0 means to turn off forwarding

# Permanently set on ip_forward function
[root@lvs1 ~]# echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf 
[root@lvs1 ~]# sysctl -p

# Test access from client to server
[root@client1 ~]# curl http://192.168.2.100/
192.168.2.100
[root@client1 ~]# curl http://192.168.2.200/
apache web server2
  • Install LVS
[root@lvs1 ~]# yum install -y ipvsadm
  • ipvsadm instructions
[root@lvs1 ~]# ipvsadm
-A: Add virtual server
-E: Edit virtual server
-D: Delete virtual server
-t: Add real tcp The server
-u: Add real udp The server
-s: Specify the scheduling algorithm. Such as polling rr/Weighted polling wrr/Minimum connection lc/Weighted least connection wlc

-a: After adding a virtual server, add a real server to the virtual server
-r: Specify the real server
-w: Set weight
-m: The specified working mode is NAT
-g: The specified working mode is DR
  • Configure LVS
# Create a virtual server for the web server and use the rr scheduling algorithm
[root@lvs1 ~]# ipvsadm -A -t 192.168.4.5:80 -s rr
# View configuration
[root@lvs1 ~]# ipvsadm -Ln

# Add RIP to virtual server
[root@lvs1 ~]# ipvsadm -a -t 192.168.4.5:80 -r 192.168.2.100 -w 1 -m
[root@lvs1 ~]# ipvsadm -a -t 192.168.4.5:80 -r 192.168.2.200 -w 2 -m
# View configuration
[root@lvs1 ~]# ipvsadm -Ln

# verification
[root@client1 ~]# for i in {1..4}
> do
> curl http://192.168.4.5/
> done
apache web server2
192.168.2.100
apache web server2
192.168.2.100

# Delete configuration. (if the configuration is wrong, use the following command to delete the reconfiguration)
[root@lvs1 ~]# ipvsadm -D -t 192.168.4.5:80


# Modify the scheduling mode to weighted polling
[root@lvs1 ~]# ipvsadm -E -t 192.168.4.5:80 -s wrr
# Verify configuration
[root@client1 ~]# for i in {1..6}; do curl http://192.168.4.5/; done
apache web server2
apache web server2
192.168.2.100
apache web server2
apache web server2
192.168.2.100

LVS DR mode

[the external link image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-TceI136X-1644236949342)(... / imgs/image-20211102151338519.png)]

  • In LVS DR mode, both LVS host and web server are single network cards. They are connected in the same network
  • Modify the experimental environment
    • client1: eth0-> 192.168.4.10
    • lvs1: eth0 - > 192.168.4.5, delete the IP address of eth1
    • web1: eth0 - > 192.168.4.100, delete the IP address of eth1
    • web2: eth0 - > 192.168.4.200, delete the IP address of eth1
# Delete lvs virtual server configuration
[root@lvs1 ~]# ipvsadm -D -t 192.168.4.5:80
[root@lvs1 ~]# ipvsadm -Ln

# Delete the configuration of eth1 on lvs1
[root@lvs1 ~]# nmcli connection modify eth1 ipv4.method disabled ipv4.addresses ''
[root@lvs1 ~]# ifdown eth1


# Modify the configuration of web1: stop the address of eth1. The address of eth0 is 192.168.4.100
# Enter the network card configuration file directory
[root@web1 ~]# cd /etc/sysconfig/network-scripts/
# The configuration file of eth0 network card is ifcfg-eth0
[root@web1 network-scripts]# ls ifcfg-eth*
ifcfg-eth0  ifcfg-eth1	ifcfg-eth2  ifcfg-eth3
# Configure eth0 address
[root@web1 network-scripts]# vim ifcfg-eth0
TYPE=Ethernet             # The network type is Ethernet
BOOTPROTO=none            # The IP address is statically configured, or static can be used
NAME=eth0                 # Rename device
DEVICE=eth0               # Network card device name
ONBOOT=yes                # Boot and activate network card
IPADDR=192.168.4.100      # IP address
NETMASK=255.255.255.0     # Subnet mask
GATEWAY=192.168.4.254     # gateway
[root@web1 network-scripts]# ifdown eth0; ifup eth0  # Disable active network card

# Stop eth1 on web1
[root@web1 ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth1
TYPE=Ethernet
BOOTPROTO=none
NAME=eth1
DEVICE=eth1
ONBOOT=no
[root@web1 ~]# ifdown eth1


# Modify the network of web2
[root@web2 ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth0 
TYPE=Ethernet
BOOTPROTO=none
NAME=eth0
DEVICE=eth0
ONBOOT=yes
IPADDR=192.168.4.200
NETMASK=255.255.255.0
GATEWAY=192.168.4.254
[root@web2 ~]# ifdown eth0; ifup eth0
[root@web2 ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth1
TYPE=Ethernet
BOOTPROTO=none
NAME=eth1
DEVICE=eth1
ONBOOT=no
[root@web2 ~]# ifdown eth1   # The terminal is stuck, turn it off and reconnect at the new terminal

Configure LVS DR mode

  1. Configure vip 192.168.4.15 on eth0 of lvs1. Configure vip by creating a logical port for eth0, and name the logical port eth0:0
[root@lvs1 ~]# cd /etc/sysconfig/network-scripts/
[root@lvs1 network-scripts]# cp ifcfg-eth0 ifcfg-eth0:0
[root@lvs1 network-scripts]# vim ifcfg-eth0:0
TYPE=Ethernet
BOOTPROTO=none
NAME=eth0:0
DEVICE=eth0:0
ONBOOT=yes
IPADDR=192.168.4.15
PREFIX=24
[root@lvs1 network-scripts]# ifup eth0:0
# View new address
[root@lvs1 network-scripts]# ifconfig 
  1. Configure vip 192.168.4.15 on the lo of two web servers
[root@web1 ~]# cd /etc/sysconfig/network-scripts/
[root@web1 network-scripts]# cp ifcfg-lo ifcfg-lo:0
[root@web1 network-scripts]# vim ifcfg-lo:0
DEVICE=lo:0
IPADDR=192.168.4.15
NETMASK=255.255.255.255
NETWORK=192.168.4.15
BROADCAST=192.168.4.15
ONBOOT=yes
NAME=lo:0
[root@web1 network-scripts]# ifup lo:0
[root@web1 network-scripts]# ifconfig 
# Copy the configuration of web1 to web2
[root@web1 network-scripts]# scp ./ifcfg-lo:0 192.168.4.200:/etc/sysconfig/network-scripts/
[root@web2 ~]# ifup lo:0
[root@web2 ~]# ifconfig 
  1. Configure the kernel parameters on the two web servers so that they do not respond to the request of 192.168.4.15
[root@web1 ~]# sysctl -a | grep arp_ignore
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 0
[root@web1 ~]# sysctl -a | grep arp_announce
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 0

[root@web1 ~]# vim /etc/sysctl.conf 
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
[root@web1 ~]# sysctl -p

[root@web2 ~]# vim /etc/sysctl.conf 
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
[root@web2 ~]# sysctl -p
  1. Configuring a virtual server on lvs1
# Create virtual server
[root@lvs1 ~]# ipvsadm -A -t 192.168.4.15:80 -s wlc
# Adding a real server to a virtual server
[root@lvs1 ~]# ipvsadm -a -t 192.168.4.15:80 -r 192.168.4.100 -w 1 -g
[root@lvs1 ~]# ipvsadm -a -t 192.168.4.15:80 -r 192.168.4.200 -w 2 -g
# View configuration
[root@lvs1 ~]# ipvsadm -Ln

# Customer verification
[root@client1 ~]# for i in {1..6}; do curl http://192.168.4.15/; done
apache web server2
192.168.2.100
apache web server2
apache web server2
192.168.2.100
apache web server2
  1. lvs itself has no application server monitoring function. If there is a problem with the web server, it needs to be deleted from the rules manually
[root@web1 ~]# systemctl stop httpd
# When the client accesses, the request forwarded to web1 will reject the connection
[root@client1 ~]# for i in {1..6}; do curl http://192.168.4.15/; done
apache web server2
curl: (7) Failed connect to 192.168.4.15:80; connection denied
apache web server2
apache web server2
curl: (7) Failed connect to 192.168.4.15:80; connection denied
apache web server2
# Delete web1 from rules
[root@lvs1 ~]# ipvsadm -d -t 192.168.4.15:80 -r 192.168.4.100
[root@client1 ~]# for i in {1..6}; do curl http://192.168.4.15/; done
apache web server2
apache web server2
apache web server2
apache web server2
apache web server2
apache web server2

Attachment: troubleshooting steps in case of error:

# The web server can be accessed on lvs
[root@lvs1 ~]# curl http://192.168.4.100/
192.168.2.100
[root@lvs1 ~]# curl http://192.168.4.200/
apache web server2

# View vip
[root@lvs1 ~]# ifconfig eth0:0
eth0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.4.15  netmask 255.255.255.0  broadcast 192.168.4.255
        ether 52:54:00:0d:fb:79  txqueuelen 1000  (Ethernet)

[root@web1 ~]# ifconfig lo:0
lo:0: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 192.168.4.15  netmask 255.255.255.255
        loop  txqueuelen 1000  (Local Loopback)

# View kernel parameters
[root@web1 ~]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2

# View rules
[root@lvs1 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.4.15:80 wlc
  -> 192.168.4.100:80             Route   1      0          0         
  -> 192.168.4.200:80             Route   2      0          0  

Keywords: Linux network server cluster

Added by Attilitus on Mon, 07 Feb 2022 23:23:05 +0200