DPVS fullnat mode deployment

This article mainly introduces in centos7 9 deploy the FullNAT mode of DPVS on the system and install the toa module on the RealServer to obtain the real IP of the client.

It has been introduced in previous articles DPVS introduction and deployment as well as Application and principle analysis of DPDK in DPVS , students in need can fill in the relevant contents first. Since the deployment steps in the previous article only introduced the deployment of DPVS and did not involve the configuration of various load balancing modes, and the version of DPVS and the corresponding version of DPDK have been updated after more than half a year, a new deployment tutorial is written here in detail.

The DPVS version installed in this article is 1.8-10 and the dpdk version is 18.11.2. Different from the above, the installation steps and operations are also different.

1. Preparatory work

After the formal installation, we need to adjust the hardware parameters of the machine. DPVS official has certain requirements for hardware (mainly because of the dpdk used at the bottom), and dpdk official gives a copy Support List , although the platforms on the support list are widely supported, in fact, Intel's hardware platform seems to be the one with the best compatibility and performance.

1.1 hardware

1.1.1 hardware parameters

  • Machine model: PowerEdge R630
  • CPU: two Intel (R) Xeon (R) CPUs e5-2630 V4 @ 2.20GHz
  • Memory: 16G*8 DDR4-2400 MT/s (Configured in 2133 MT/s), 64g for each CPU64G, 128G in total
  • Network card 1: Intel Corporation 82599es 10 Gigabit SFI / SFP + network connection (Rev 01)
  • Network card 2: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
  • System: CentOS Linux release 7.9.2009 (Core)
  • Kernel: 3.10.0-1160.36.2 el7. x86_ sixty-four

1.1.2 BIOS settings

Before starting, first enter the BIOS, turn off hyper threading and enable NUMA policy. DPVS is a typical CPU busy application (the CPU utilization of the process is always 100%). In order to ensure performance, it is recommended to turn off the hyper threading setting of the CPU. In order to ensure the affinity of the CPU, it is best to open the page manually in the BIOS for the sake of the affinity of the CPU.

1.1.3 network card PCI ID

After using the PMD driver of dpvs to take over the network card, if the number of network cards is large and easy to be confused, it is best to record the corresponding network card name, MAC address and PCI ID in advance to avoid confusion in subsequent operations.

Use the lspci command to view the PCI ID of the corresponding network card. Then we can view the device file under the corresponding network card name directory in the directory / sys/class/net / to know the PCI ID of the network card. Finally, you can string the network card name, MAC address and PCI ID.

$ lspci | grep -i net
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

$ file /sys/class/net/eth0/device
/sys/class/net/eth0/device: symbolic link to `../../../0000:01:00.0'

1.2 software

1.2.1 system software

# Tools for compiling and installing dpvs and tools for viewing CPU NUMA information
$ yum group install "Development Tools" 
$ yum install patch libnuma* numactl numactl-devel kernel-devel openssl* popt* libpcap-devel -y
# If you need ipvsadm to support ipv6, you need to install libnl3 devel
$ yum install libnl libnl-devel libnl3 libnl3-devel -y

# Note that the version of the kernel and the corresponding kernel components should correspond to the current version of the kernel
$ uname -r
$ rpm -qa | grep kernel | grep "3.10.0-1160.36.2"

1.2.2 dpvs and dpdk

# dpvs we directly use git to pull the latest version from github
$ git clone https://github.com/iqiyi/dpvs.git
# dpdk we download version 18.11.2 from the official website and put it in the dpvs directory for easy operation
$ cd dpvs/
$ wget https://fast.dpdk.org/rel/dpdk-18.11.2.tar.xz
$ tar -Jxvf dpdk-18.11.2.tar.xz

After completing the above steps, you can start the following installation.

2. Installation steps

2.1 DPDK installation

2.1.1 installing dpdk patch

Under the patch directory of the dpvs folder, there are patches of the corresponding supported dpdk version. If you don't know which patch you need, the official recommendation is to install all of them

$ ll dpvs/patch/dpdk-stable-18.11.2
total 44
-rw-r--r-- 1 root root  4185 Jul 22 12:47 0001-kni-use-netlink-event-for-multicast-driver-part.patch
-rw-r--r-- 1 root root  1771 Jul 22 12:47 0002-net-support-variable-IP-header-len-for-checksum-API.patch
-rw-r--r-- 1 root root  1130 Jul 22 12:47 0003-driver-kni-enable-flow_item-type-comparsion-in-flow_.patch
-rw-r--r-- 1 root root  1706 Jul 22 12:47 0004-rm-rte_experimental-attribute-of-rte_memseg_walk.patch
-rw-r--r-- 1 root root 16538 Jul 22 12:47 0005-enable-pdump-and-change-dpdk-pdump-tool-for-dpvs.patch
-rw-r--r-- 1 root root  2189 Jul 22 12:47 0006-enable-dpdk-eal-memory-debug.patch

The operation of installing patch is also very simple

# We first copy all the patch es to the root directory of dpdk
$ cp dpvs/patch/dpdk-stable-18.11.2/*patch dpvs/dpdk-stable-18.11.2/
$ cd dpvs/dpdk-stable-18.11.2/
# Then we install it in the order of the file names of the patch
$ patch -p 1 < 0001-kni-use-netlink-event-for-multicast-driver-part.patch
patching file kernel/linux/kni/kni_net.c
$ patch -p 1 < 0002-net-support-variable-IP-header-len-for-checksum-API.patch
patching file lib/librte_net/rte_ip.h
$ patch -p 1 < 0003-driver-kni-enable-flow_item-type-comparsion-in-flow_.patch
patching file drivers/net/mlx5/mlx5_flow.c
$ patch -p 1 < 0004-rm-rte_experimental-attribute-of-rte_memseg_walk.patch
patching file lib/librte_eal/common/eal_common_memory.c
Hunk #1 succeeded at 606 (offset 5 lines).
patching file lib/librte_eal/common/include/rte_memory.h
$ patch -p 1 < 0005-enable-pdump-and-change-dpdk-pdump-tool-for-dpvs.patch
patching file app/pdump/main.c
patching file config/common_base
patching file lib/librte_pdump/rte_pdump.c
patching file lib/librte_pdump/rte_pdump.h
$ patch -p 1 < 0006-enable-dpdk-eal-memory-debug.patch
patching file config/common_base
patching file lib/librte_eal/common/include/rte_malloc.h
patching file lib/librte_eal/common/rte_malloc.c

2.1.2 dpdk compilation and installation

$ cd dpvs/dpdk-stable-18.11.2
$ make config T=x86_64-native-linuxapp-gcc
$ make 

# The words "build complete [x86_64 native Linux app GCC] appear to indicate that make is successful

$ export RTE_SDK=$PWD
$ export RTE_TARGET=build

Dpdk17.0 will not appear in the process of compiling and installing here Ndo in version 11.2_ change_ MTU problem

2.1.3 configure hugepage

Different from other general programs, the dpdk used by dpvs does not ask for memory from the operating system, but directly uses large page memory, which greatly improves the efficiency of memory allocation. The configuration of hugepage is relatively simple. The official configuration process uses 2MB of large page memory. 28672 here refers to 28672 2MB of large page memory allocated, that is, 56GB of memory corresponding to a node. A total of 112GB of memory is allocated. The memory here can be adjusted according to the size of the machine. However, if it is less than 1GB, it may cause startup error.

A single CPU system can refer to dpdk Official documents

# for NUMA machine
$ echo 28672 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
$ echo 28672 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages

$ mkdir /mnt/huge
$ mount -t hugetlbfs nodev /mnt/huge

# If you need to start up and mount automatically, you can
$ echo "nodev /mnt/huge hugetlbfs defaults 0 0" >> /etc/fstab

# After the configuration is completed, we can see that the memory utilization rate increases immediately
$ free -g	# Before configuration
              total        used        free      shared  buff/cache   available
Mem:            125           1         122           0           1         123
$ free -g	# After configuration
              total        used        free      shared  buff/cache   available
Mem:            125         113          10           0           1          11
# Using numactl to check the memory status, you can also see that 56G of CPU memory is allocated on both sides
$ numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14 16 18
node 0 size: 64184 MB
node 0 free: 4687 MB
node 1 cpus: 1 3 5 7 9 11 13 15 17 19
node 1 size: 64494 MB
node 1 free: 5759 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10

2.1.4 configuring ulimit

By default, the ulimit of the system limits the number of open file descriptors. If it is too small, it will affect the normal operation of dpvs, so we increase it:

$ ulimit -n 655350
$ echo "ulimit -n 655350" >> /etc/rc.local
$ chmod a+x /etc/rc.local

2.2 mount the drive module

First, we need to let the system mount the dpdk driver (PMD driver) we have compiled, and then replace the default driver used by the network card with the PMD driver we have compiled here

$ modprobe uio
$ insmod /path/to/dpdk-stable-18.11.2/build/kmod/igb_uio.ko
$ insmod /path/to/dpdk-stable-18.11.2/build/kmod/rte_kni.ko carrier=on

It should be noted that the carrier parameter is from dpdk v18 The default value is off when the version 11 is added. We need to load RTE_ KNI. The KNI equipment can work normally only when the carrier=on parameter is brought when the KO module is.

In the dpdk-stable-18.11.2/usertools directory, there are some scripts to help us install and use dpdk. We can use them to reduce the complexity of configuration. Here we can use dpdk devbind Py script to change the driver of the network card

# First, we turn off the network card that we need to load the PMD driver
$ ifdown eth{2,3,4,5}

# Check the status of the network card. Pay special attention to the PCI ID corresponding to the network card. Below, only some useful output results are intercepted
$ ./usertools/dpdk-devbind.py --status
Network devices using kernel driver
0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=eth2 drv=ixgbe unused=igb_uio
0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=eth3 drv=ixgbe unused=igb_uio
0000:82:00.0 'Ethernet 10G 2P X520 Adapter 154d' if=eth4 drv=ixgbe unused=igb_uio
0000:82:00.1 'Ethernet 10G 2P X520 Adapter 154d' if=eth5 drv=ixgbe unused=igb_uio

From the above output results, we can see that the current network card uses ixgbe driver, and our goal is to make it use igb_uio driver. Note that if there are too many network cards in the system at this time, the three parameters of network card name MAC address PCI ID recorded earlier can be used.

# Load specific drivers for network cards that need to use dpvs
$ ./usertools/dpdk-devbind.py -b igb_uio 0000:04:00.0
$ ./usertools/dpdk-devbind.py -b igb_uio 0000:04:00.1
$ ./usertools/dpdk-devbind.py -b igb_uio 0000:82:00.0
$ ./usertools/dpdk-devbind.py -b igb_uio 0000:82:00.1

# Check whether the loading is successful again. Only some useful output results are intercepted below
$ ./usertools/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=igb_uio unused=ixgbe
0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=igb_uio unused=ixgbe
0000:82:00.0 'Ethernet 10G 2P X520 Adapter 154d' drv=igb_uio unused=ixgbe
0000:82:00.1 'Ethernet 10G 2P X520 Adapter 154d' drv=igb_uio unused=ixgbe

2.3 DPVS installation

$ cd /path/to/dpdk-stable-18.11.2/
$ export RTE_SDK=$PWD
$ cd /path/to/dpvs
$ make 
$ make install
# View the binary files in the bin directory
$ ls /path/to/dpvs/bin/
dpip  dpvs  ipvsadm  keepalived

# Pay attention to the prompt information in the make process, especially the kept part. If the following part appears, it means that IPVS supports IPv6
Keepalived configuration
Keepalived version       : 2.0.19
Compiler                 : gcc
Preprocessor flags       : -D_GNU_SOURCE -I/usr/include/libnl3
Compiler flags           : -g -g -O2 -fPIE -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -O2
Linker flags             : -pie -Wl,-z,relro -Wl,-z,now
Extra Lib                : -lm -lcrypto -lssl -lnl-genl-3 -lnl-3
Use IPVS Framework       : Yes
IPVS use libnl           : Yes
IPVS syncd attributes    : No
IPVS 64 bit stats        : No

# In order to facilitate management, relevant operation commands can be soft linked to / sbin for global execution
$ ln -s /path/to/dpvs/bin/dpvs /sbin/dpvs
$ ln -s /path/to/dpvs/bin/dpip /sbin/dpip
$ ln -s /path/to/dpvs/bin/ipvsadm /sbin/ipvsadm
$ ln -s /path/to/dpvs/bin/keepalived /sbin/keepalived

# Check whether the commands related to dpvs can work normally. Note that other commands can be used normally only after the dpvs process is started
$ dpvs -v
dpvs version: 1.8-10, build on 2021.07.26.15:34:26

2.4 configuring DPVS conf

Under the dpvs/conf directory, there are examples of dpvs configuration files with various configuration modes, and at the same time, in dpvs All parameters are recorded in the conf.items file. It is recommended that students read them all and understand the basic syntax before configuring them. The default configuration file for dpvs startup is / etc / dpvs conf.

Here is a brief summary of some parts (! Is a comment symbol):

  • The format of the log can be manually adjusted to DEBUG, and the location of the log output can be modified to facilitate the location of the problem

    global_defs {
        log_level   DEBUG
        log_file    /path/to/dpvs/logs/dpvs.log
  • If you need to define multiple network cards, you can refer to this configuration

    netif_defs {
        <init> pktpool_size     1048575
        <init> pktpool_cache    256
        <init> device dpdk0 {
            rx {
                queue_number        16
                descriptor_number   1024
                rss                 all
            tx {
                queue_number        16
                descriptor_number   1024
            fdir {
                mode                perfect
                pballoc             64k
                status              matched
            kni_name                dpdk0.kni
        <init> device dpdk1 {
            rx {
                queue_number        16
                descriptor_number   1024
                rss                 all
            tx {
                queue_number        16
                descriptor_number   1024
            fdir {
                mode                perfect
                pballoc             64k
                status              matched
            kni_name                dpdk1.kni
        <init> device dpdk2 {
            rx {
                queue_number        16
                descriptor_number   1024
                rss                 all
            tx {
                queue_number        16
                descriptor_number   1024
            fdir {
                mode                perfect
                pballoc             64k
                status              matched
            kni_name                dpdk2.kni
        <init> device dpdk3 {
            rx {
                queue_number        16
                descriptor_number   1024
                rss                 all
            tx {
                queue_number        16
                descriptor_number   1024
            fdir {
                mode                perfect
                pballoc             64k
                status              matched
            kni_name                dpdk3.kni
  • The same transceiver queue of multiple network cards shares the same CPU

        <init> worker cpu1 {
            type    slave
            cpu_id  1
            port    dpdk0 {
                rx_queue_ids     0
                tx_queue_ids     0
            port    dpdk1 {
                rx_queue_ids     0
                tx_queue_ids     0
            port    dpdk2 {
                rx_queue_ids     0
                tx_queue_ids     0
            port    dpdk3 {
                rx_queue_ids     0
                tx_queue_ids     0
  • If you need to specify a CPU separately to process ICMP packets, you can add ICMP to the parameters of the worker_ redirect_ core

        <init> worker cpu16 {
            type    slave
            cpu_id  16
            port    dpdk0 {
                rx_queue_ids     15
                tx_queue_ids     15

After the DPVS process is started, the corresponding network card can be configured directly in the network configuration file of the Linux system, which is exactly the same as other network cards such as eth0.

After running successfully, you can see the corresponding dpdk network card by using the dpip command and the normal IP and ifconfig commands, and the IPv4 and IPv6 networks can be used normally. The following figure only intercepts part of the information, the IP and MAC information has been desensitized, and the IPv6 information has been removed.

$ dpip link show
1: dpdk0: socket 0 mtu 1500 rx-queue 16 tx-queue 16
    UP 10000 Mbps full-duplex auto-nego

$ ip a
67: dpdk0.kni: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether AA:BB:CC:23:33:33 brd ff:ff:ff:ff:ff:ff
    inet brd scope global dpdk0.kni
       valid_lft forever preferred_lft forever
$ ifconfig dpdk0.kni
dpdk0.kni: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet  netmask  broadcast
        ether AA:BB:CC:23:33:33  txqueuelen 1000  (Ethernet)
        RX packets 1790  bytes 136602 (133.4 KiB)
        RX errors 0  dropped 52  overruns 0  frame 0
        TX packets 115  bytes 24290 (23.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

3. Configure FullNat

In order to verify that our DPVS can work normally, here we refer to the official Configuration document First, configure an FNAT with the simplest dual arm mode. Referring to the official architecture diagram and modifying the IP address information, we can get the following simple architecture diagram.

In this mode, it is not necessary to configure the kni network card virtualized by DPVS by using the tools such as ip and ifconfig provided by the system

[the external chain image transfer fails. The source station may have anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-ywrjxom5-1645782758324)( https://resource.tinychen.com/20210728123202.svg )]

Here, we use the dpdk2 network card as the wan port and the dpdk0 network card as the lan port

# First, we add VIP to the dpdk2 network card (wan)
$ dpip addr add dev dpdk2

# Next, we need to add two routes, which are divided into wan port network segment and RS machine network segment
$ dpip route add dev dpdk2
$ dpip route add dev dpdk0
# It is better to add a default route to the gateway to ensure that the return packet of ICMP packet can run through
$ dpip route add default via dev dpdk2

# Establish forwarding rules using RR algorithm
# add service <VIP:vport> to forwarding, scheduling mode is RR.
# use ipvsadm --help for more info.
$ ipvsadm -A -t -s rr

# Here, for the convenience of testing, we only add one RS
# add two RS for service, forwarding mode is FNAT (-b)
$ ipvsadm -a -t -r -b

# Add LocalIP to the network. FNAT mode is required here
# add at least one Local-IP (LIP) for FNAT on LAN interface
$ ipvsadm --add-laddr -z -t -F dpdk0

# Then let's see the effect
$ dpip route show
inet via src dev dpdk0 mtu 1500 tos 0 scope host metric 0 proto auto
inet via src dev dpdk2 mtu 1500 tos 0 scope host metric 0 proto auto
inet via src dev dpdk2 mtu 1500 tos 0 scope link metric 0 proto auto
inet via src dev dpdk0 mtu 1500 tos 0 scope link metric 0 proto auto
inet via src dev dpdk2 mtu 1500 tos 0 scope global metric 0 proto auto

$ dpip addr show
inet scope global dpdk2
     valid_lft forever preferred_lft forever
inet scope global dpdk0
     valid_lft forever preferred_lft forever

$ ipvsadm  -ln
IP Virtual Server version 0.0.0 (size=0)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP rr
  ->              FullNat 1      0          0
$ ipvsadm  -G
VIP:VPORT            TOTAL    SNAT_IP              CONFLICTS  CONNS    1
                           0          0

Then we start an nginx on RS and set the return IP and port number to see the effect:

    server {
        listen 80 default;

        location / {
            default_type text/plain;
            return 200 "Your IP and port is $remote_addr:$remote_port\n";


Test VIP directly with ping and curl commands:

$ ping -c4
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=54 time=47.2 ms
64 bytes from icmp_seq=2 ttl=54 time=48.10 ms
64 bytes from icmp_seq=3 ttl=54 time=48.5 ms
64 bytes from icmp_seq=4 ttl=54 time=48.5 ms

--- ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 8ms
rtt min/avg/max/mdev = 47.235/48.311/48.969/0.684 ms

$ curl
Your IP and port is

It can be found that no matter what machine it is on, only the IP and port number of LIP will be returned. If you need to obtain the user's real IP, you need to install the TOA module

4. RS installation TOA module

At present, there are many versions of TOA modules provided by the open source community. Here, in order to ensure compatibility, we directly use the TOA and uoa modules officially provided by dpvs. According to their official description, their toa modules are separated from Alibaba TOA

TOA source code is included into DPVS project(in directory kmod/toa) since v1.7 to support IPv6 and NAT64. It is derived from the Alibaba TOA. For IPv6 applications which need client's real IP address, we suggest to use this TOA version.

Since both RS machines and DPVS machines here use the CentOS7 system, we can directly compile the toa module on the DPVS machine and then copy it to each RS machine for use

$ cd /path/to/dpvs/kmod/toa/
$ make

After successful compilation, a toa will be generated in the current directory Ko module file, which is the file we need. Directly use the insmod command to load the module and then check it

$ insmod toa.ko
$ lsmod  | grep toa
toa                   279641  0

Ensure that the module can be loaded at RC Add the following instructions to the local file

/usr/sbin/insmod /path/to/toa.ko
# for example: 
# /usr/sbin/insmod /home/dpvs/kmod/toa/toa.ko

In addition to toa module, there is also uoa module for UDP protocol, which is completely consistent with the compilation and installation process of TOA module above. It will not be repeated here.

After we load the curs module on the machine again:

$ curl
Your IP and port is

So far, the FullNat mode of the whole DPVS has been deployed and can work normally. Since DPVS supports many configuration combinations, a special article on IPv6, nat64, keepalived, bonding and Master/Backup mode configuration will be written later.

Keywords: Load Balance lvs

Added by DeathStar on Sat, 26 Feb 2022 18:54:28 +0200