1. General
As the longest used network plug-in of Kubernetes, Calico has strong scalability, better resource utilization and less dependence. Compared with the Overlay network adopted by Flannel plug-in, Calico can adopt the Underlay network with better performance through three-layer routing. The forwarding efficiency of Calico network plug-in is higher than that of all schemes.
In the actual production environment using calico network plug-in, in order to improve the performance and flexibility of the network, it is necessary to establish BGP neighbor relationship between the working node of K8S and the leaf switch in the physical network, synchronize BGP routing information, and publish the routing of pod network to the physical network. Calico provides three types of BGP interconnection schemes, namely, full mesh, Route reflectors and Top of Rack (ToR).
Full-mesh
In the fully interconnected mode, after BGP is enabled, Calico's default behavior is to create a complete internal BGP (iBGP) connection when each node is peer-to-peer, which enables Calico to run on any L2 network (whether public cloud or private cloud), or (if equipped with IPIP) as an overlay on any network that does not prohibit IPIP traffic. For vxlan overlay, Calico does not use BGP.
The full mesh mode is very useful for small and medium-sized deployments with less than 100 working nodes or fewer nodes, but the efficiency of the full mesh mode will be reduced on a large scale. In the case of a large scale, Calico officially recommends the use of Route reflectors.
Route reflectors
If you want to build a large-scale internal BGP (iBGP) cluster, you can use BGP routing reflector to reduce the number of BGP peers on each node. In this model, some nodes act as routing reflectors and are configured to establish a complete grid between them. Then, other nodes are configured to peer with a subset of these routing reflectors (usually redundant, usually 2), thereby reducing the total number of BGP peer-to-peer connections compared with the whole grid.
Top of Rack(ToR)
In a local deployment, Calico can be configured to peer directly to the physical network infrastructure. Typically, this involves disabling Calico's default full mesh behavior and peer all Calico nodes to the L3 ToR router.
This article focuses on how to configure Calico routing reflector in BGP network environment. This article mainly introduces how to use K8S working node as routing reflector to establish BGP connection with physical switch. The configuration environment topology is as follows:
In this environment, there is one spin switch and two leaf switches to establish EBGP connection. All leaf switches belong to an independent autonomous system, and nodes under all leaf switches belong to an independent autonomous system. In each leaf of Kubernetes cluster node, two working nodes are used as CalicoRR (routing reflectors). The reason why two nodes are used as routing reflectors is to consider redundancy. All calico RRs establish EBGP connections with their associated leaf switches. Establish an iBGP connection between Calico RR and its node.
Install calicoctl
All configuration operations of Calico RR need to be completed through calicoctl tool. Calicoctl allows the creation, reading, updating and deletion of Calico objects from commands. Therefore, we first need to install calicoctl tool on all work nodes of Kubernetes.
Install the calicoctl tool in binary mode.
Log in to the host, open the terminal prompt, and then navigate to the installation binary file location. Generally, calicoctl is installed to / usr/local/bin /.
Use the following command to download the calicoctl binary file, and select your own calico version for the version number.
$ curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.17.2/calicoctl
Command use
[root@node1 ~]# calicoctl -h Usage: calicoctl [options] <command> [<args>...] create Create a resource by filename or stdin. replace Replace a resource by filename or stdin. apply Apply a resource by filename or stdin. This creates a resource if it does not exist, and replaces a resource if it does exists. patch Patch a pre-exisiting resource in place. delete Delete a resource identified by file, stdin or resource type and name. get Get a resource identified by file, stdin or resource type and name. label Add or update labels of resources. convert Convert config files between different API versions. ipam IP address management. node Calico node management. version Display the version of calicoctl. export Export the Calico datastore objects for migration import Import the Calico datastore objects for migration datastore Calico datastore management. Options: -h --help Show this screen. -l --log-level=<level> Set the log level (one of panic, fatal, error, warn, info, debug) [default: panic] Description: The calicoctl command line tool is used to manage Calico network and security policy, to view and manage endpoint configuration, and to manage a Calico node instance. See 'calicoctl <command> --help' to read about a specific subcommand.
Turn off full mesh mode
Calico defaults to the full mesh full interconnection mode. All nodes in the calico cluster will establish connections for routing exchange. However, with the expansion of cluster scale, mesh mode will form a huge service grid, and the number of connections will increase exponentially. At this time, you need to use the Route Reflector mode to solve this problem. Determine one or more calico nodes to act as routing reflectors, and let other nodes obtain routing information from this RR node.
Close the node to node BGP network. The specific steps are as follows:
Add the default BGP configuration and adjust nodeToNodeMeshEnabled and asNumber:
[root@node1 calico]# cat bgpconf.yaml apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: logSeverityScreen: Info nodeToNodeMeshEnabled: false asNumber: 64512
Apply it directly, and full mesh will be disabled immediately after application,
[root@node1 calico]# calicoctl apply -f bgpconf.yaml Successfully applied 1 'BGPConfiguration' resource(s)
Check the bgp network configuration. false means off
[root@node1 calico]# calicoctl get bgpconfig NAME LOGSEVERITY MESHENABLED ASNUMBER default Info false 64512
Modify calico configuration of work node
The ASN number of each node can be obtained through calicoctl get nodes – output=wide,
[root@node1 calico]# calicoctl get nodes --output=wide NAME ASN IPV4 IPV6 node1 (64512) 172.20.0.11/24 node2 (64512) 172.20.0.12/24 node3 (64512) 172.20.0.13/24 node4 (64512) 173.20.0.11/24 node5 (64512) 173.20.0.12/24 node6 (64512) 173.20.0.13/24
You can see that the ASN number obtained is "(64512)", because if you do not specify an ASN number for each node, it is 64512 by default. We can configure the ASN number of each node according to the topology. The ASN number of nodes under different leaf switches is different. The working node under each leaf switch is an independent autonomous system.
Obtain the calico configuration information of the work node through the following command:
$ calicoctl get node node1 -o yaml > node1.yaml
The calico configuration information of each work node needs to be obtained. The output is yaml file, "node1" is the name of calico node
Modify according to the following format:
[root@node1 calico]# cat node1.yaml apiVersion: projectcalico.org/v3 kind: Node metadata: annotations: projectcalico.org/kube-labels: '{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"node1","kubernetes.io/os":"linux","node-role.kubernetes.io/master":"","node-role.kubernetes.io/worker":"","rr-group":"rr1","rr-id":"rr1"}' creationTimestamp: null labels: beta.kubernetes.io/arch: amd64 beta.kubernetes.io/os: linux kubernetes.io/arch: amd64 kubernetes.io/hostname: node1 kubernetes.io/os: linux node-role.kubernetes.io/master: "" node-role.kubernetes.io/worker: "" name: node1 spec: bgp: asNumber: 64512 ## asNumber can be modified as needed ipv4Address: 172.20.0.11/24 routeReflectorClusterID: 172.20.0.11 ## routeReflectorClusterID is generally changed to the IP address of its own node orchRefs: - nodeName: node1 orchestrator: k8s status: podCIDRs: - "" - 10.233.64.0/24
After modifying the Calico configuration information of all nodes, the node information obtained through the calicoctl get nodes -o wide command is as follows:
[root@node1 calico]# calicoctl get nodes -o wide NAME ASN IPV4 IPV6 node1 64512 172.20.0.11/24 node2 64512 172.20.0.12/24 node3 64512 172.20.0.13/24 node4 64513 173.20.0.11/24 node5 64513 173.20.0.12/24 node6 64513 173.20.0.13/24
As can be seen from the above, all ASN s have been manually specified, which is no longer the global default.
Group node nodes (add label)
In order to make it easy for BGPPeer to select nodes, in Kubernetes cluster, we need to group all nodes by labeling. Here, we divide the label into the following types:
RR group is defined here as the Calico RR group to which the node belongs, mainly including rr1 and rr2. It is the Calico RR under different leaf switches. RR ID is defined here as the ID of the Calico RR to which the node belongs. Adding this label to the node indicates that the node acts as a routing reflector, mainly including rr1 and rr2. It is the Calico RR under different leaf switches
Add a label for each node with the following command:,
$ kubectl label nodes node1 rr-group=rr1
$ kubectl label nodes node1 rr-id=rr1
Check the final settings,
[root@node1 calico]# kubectl get nodes --show-labels NAME STATUS ROLES AGE VERSION LABELS node1 Ready master,worker 31d v1.17.9 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node1,kubernetes.io/os=linux,node-role.kubernetes.io/master=,node-role.kubernetes.io/worker=,rr-group=rr1,rr-id=rr1 node2 Ready master,worker 31d v1.17.9 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node2,kubernetes.io/os=linux,node-role.kubernetes.io/master=,node-role.kubernetes.io/worker=,rr-group=rr1,rr-id=rr1 node3 Ready master,worker 31d v1.17.9 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node3,kubernetes.io/os=linux,node-role.kubernetes.io/master=,node-role.kubernetes.io/worker=,rr-group=rr1 node4 Ready worker 16d v1.17.9 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node4,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,rr-group=rr2,rr-id=rr2 node5 Ready worker 16d v1.17.9 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node5,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,rr-group=rr2,rr-id=rr2 node6 Ready worker 16d v1.17.9 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node5,kubernetes.io/os=linux,node-role.kubernetes.io/worker=,rr-group=rr2,rr-id=rr2
Configure BGPPeer
Before configuring BGPPeer, we can check the node status of each node BGP. Because full mesh has been disabled and BGPPeer has not been configured yet, the information in all nodes is empty.
[root@node3 ~]# calicoctl node status Calico process is running. IPv4 BGP status No IPv4 peers found. IPv6 BGP status No IPv6 peers found.
According to the environment topology, node1 and node2, as Calico RR, need to establish BGP connection with leaf01 switch; node4 and node5, as Calico RR, need to establish BGP connection with leaf02 switch; Node1, node2 and node3 need to establish BGP connection with RR1; node4, node5 and node6 need to establish BGP connection with RR2. Follow the steps below:
- Establish BGP connection between RR1 and leaf01
Write a configuration file named "rr1-to-leaf1-peer.yaml". Edit the configuration file as follows:
[root@node1 calico]# cat rr1-to-leaf1-peer.yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: rr1-to-leaf1-peer ## Give BGPPeer a name for easy identification spec: nodeSelector: rr-id == 'rr1' ## Add RR ID through node selector== 'rr1'Node of label peerIP: 172.20.0.254 ## Address of leaf01 switch asNumber: 65009 ## AS number of leaf01 switch
Apply this configuration,
[root@node1 calico]# calicoctl apply -f rr1-to-leaf1-peer.yaml Successfully applied 1 'BGPPeer' resource(s)
1.RR1 establishes BGP connection with its own node
The nodes of RR1 mainly include node1, node2 and node3, that is, the nodes labeled RR group = RR1. The configuration file is written as follows:
[root@node1 calico]# cat rr1-to-node-peer.yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: rr1-to-node-peer ## Give BGPPeer a name for easy identification spec: nodeSelector: rr-group == 'rr1' ## Add RR group through node selector== 'rr1'Node of label peerSelector: rr-id == 'rr1' ## Add a routing reflector with RR id = = 'rr1' label through the peer selector
Apply this configuration,
[root@node1 calico]# calicoctl apply -f rr1-to-node-peer.yaml Successfully applied 1 'BGPPeer' resource(s)
2. Operate on the leaf01 switch and establish the BGP connection between the leaf01 switch and RR1. After the switch is configured, you can view the connection status of the switch bgp peer
[leaf01]show bgp peer ipv4 BGP local router ID: 2.2.2.2 Local AS number: 65009 Total number of peers: 3 Peers in established state: 3 * - Dynamically created peer Peer AS MsgRcvd MsgSent OutQ PrefRcv Up/Down State 100.0.0.1 65008 1696 1677 0 8 23:52:28 Established 172.20.0.11 64512 1648 1506 0 4 23:51:50 Established 172.20.0.12 64512 1647 1659 0 4 23:51:44 Established
172.20.0.11 and 172.20.0.12 above are node1 and node2 nodes, that is, RR1. The status displayed as "Established" indicates that the BGP connection has been Established.
3. Establish BGP connection between RR2 and leaf02
Write a configuration file named "rr2-to-leaf2-peer.yaml". Edit the configuration file as follows:
[root@node1 calico]# cat rr2-to-leaf2-peer.yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: rr2-to-leaf2-peer ## Give BGPPeer a name for easy identification spec: nodeSelector: rr-id == 'rr2' ## Add RR ID through node selector== 'rr2'Node of label peerIP: 173.20.0.254 ## Address of leaf02 switch asNumber: 65010 ## AS number of leaf02 switch
Apply this configuration,
[root@node1 calico]# calicoctl apply -f rr2-to-leaf2-peer.yaml Successfully applied 1 'BGPPeer' resource(s)
4.RR2 establishes BGP connection with its own node
The nodes of RR2 mainly include node4, node5 and node6, that is, the nodes labeled RR group = RR2. The configuration file is written as follows:
[root@node1 calico]# cat rr2-to-node-peer.yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: rr2-to-node-peer ## Give BGPPeer a name for easy identification spec: nodeSelector: rr-group == 'rr2' ## Add RR group through node selector== 'rr2'Node of label peerSelector: rr-id == 'rr2' ## Add a routing reflector with RR id = = 'rr2' label through the peer selector
Apply this configuration,
[root@node1 calico]# calicoctl apply -f rr2-to-node-peer.yaml Successfully applied 1 'BGPPeer' resource(s)
5. Operate on the leaf02 switch and establish the BGP connection between the leaf02 switch and RR2
After the switch configuration is completed, you can view the connection status of the switch bgp peer
<leaf02>sys System View: return to User View with Ctrl+Z. [leaf02]show bgp peer ipv4 BGP local router ID: 3.3.3.3 Local AS number: 65010 Total number of peers: 3 Peers in established state: 3 * - Dynamically created peer Peer AS MsgRcvd MsgSent OutQ PrefRcv Up/Down State 100.0.0.5 65008 1561 1686 0 11 24:01:03 Established 173.20.0.11 64513 1655 1650 0 2 23:59:44 Established 173.20.0.12 64513 1661 1883 0 2 23:59:56 Established
173.20.0.11 and 173.20.0.12 above are node4 and node5 nodes, that is, RR2. The status displayed as "Established" indicates that the BGP connection has been Established.
Finally, we can view all BGPPeer configuration entries through the calicoctl get bgppeer command,
[root@node1 calico]# calicoctl get bgppeer NAME PEERIP NODE ASN rr1-to-leaf1-peer 172.20.0.254 rr-id == 'rr1' 65009 rr1-to-node-peer rr-group == 'rr1' 0 rr2-to-leaf2-peer 173.20.0.254 rr-id == 'rr2' 65010 rr2-to-node-peer rr-group == 'rr2' 0
If you want to delete a BGPPeer entry, you can use the following command
$ calicoctl delete bgppeer rr2-to-node-peer
Work node configuration verification
So far, BGPPeer configuration has been completed. You can view BGPPeer status information in each node again
Operate on node1 node
[root@node1 calico]# calicoctl node status Calico process is running. IPv4 BGP status +--------------+---------------+-------+------------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+------------+-------------+ | 172.20.0.12 | node specific | up | 2021-02-16 | Established | | 172.20.0.13 | node specific | up | 2021-02-16 | Established | | 172.20.0.254 | node specific | up | 2021-02-16 | Established | +--------------+---------------+-------+------------+-------------+ IPv6 BGP status No IPv6 peers found.
It can be seen that the node has established BGP connection with leaf01 switch, node2 and node3 nodes.
Operate on node2 node
[root@node2 ~]# calicoctl node status Calico process is running. IPv4 BGP status +--------------+---------------+-------+------------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+------------+-------------+ | 172.20.0.11 | node specific | up | 2021-02-16 | Established | | 172.20.0.13 | node specific | up | 2021-02-16 | Established | | 172.20.0.254 | node specific | up | 2021-02-16 | Established | +--------------+---------------+-------+------------+-------------+ IPv6 BGP status No IPv6 peers found.
You can see that this node has established BGP connection with leaf01 switch, node1 and node3 nodes.
Operate on node3 node
[root@node3 ~]# calicoctl node status Calico process is running. IPv4 BGP status +--------------+---------------+-------+------------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+------------+-------------+ | 172.20.0.11 | node specific | up | 2021-02-16 | Established | | 172.20.0.12 | node specific | up | 2021-02-16 | Established | +--------------+---------------+-------+------------+-------------+ IPv6 BGP status No IPv6 peers found.
It can be seen that the node has established BGP connection with node1 and node2 nodes. Because the node is not used as a routing reflector node, it also establishes BGP connection with leaf01 switch.
Operate on node4 node
[root@node4 ~]# calicoctl node status Calico process is running. IPv4 BGP status +--------------+---------------+-------+------------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+------------+-------------+ | 173.20.0.12 | node specific | up | 2021-02-16 | Established | | 173.20.0.13 | node specific | up | 2021-02-16 | Established | | 173.20.0.254 | node specific | up | 2021-02-16 | Established | +--------------+---------------+-------+------------+-------------+ IPv6 BGP status No IPv6 peers found.
You can see that this node has established BGP connection with leaf02 switch, node5 and node6 nodes.
Operate on node5 node
[root@node5 ~]# calicoctl node status Calico process is running. IPv4 BGP status +--------------+---------------+-------+------------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+------------+-------------+ | 173.20.0.11 | node specific | up | 2021-02-16 | Established | | 173.20.0.13 | node specific | up | 2021-02-16 | Established | | 173.20.0.254 | node specific | up | 2021-02-16 | Established | +--------------+---------------+-------+------------+-------------+ IPv6 BGP status No IPv6 peers found.
You can see that this node has established BGP connection with leaf02 switch, node4 and node6 nodes.
Operate on node6 node
[root@node6 ~]# calicoctl node status Calico process is running. IPv4 BGP status +--------------+---------------+-------+------------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+------------+-------------+ | 173.20.0.11 | node specific | up | 2021-02-16 | Established | | 173.20.0.12 | node specific | up | 2021-02-16 | Established | +--------------+---------------+-------+------------+-------------+ IPv6 BGP status No IPv6 peers found.
It can be seen that the node has established BGP connection with node4 and node5 nodes. Because the node is not a routing reflector node, it also establishes BGP connection with leaf02 switch.
Switch configuration verification
We can check the BGP synchronized routing information in all switches to see if there is a routing address deployed in the pod
Spine switch operation
Leaf01 switch operation
Leaf02 switch operation
In the above screenshot of switch operation, the address segments beginning with 10.233 are the routing information of the pod address segment.
Sometimes, not only the Pod address can be routed in the current network, but also the Service address. We can synchronize the route of the Service address by modifying the bgpconfig configuration,
First, check whether there is a default BGP configuration
[root@node1 ~]# calicoctl get bgpconfig default NAME LOGSEVERITY MESHENABLED ASNUMBER default Info false 64512
The default BGP configuration exists. Update the BGP configuration
[root@node1 ~]# calicoctl patch BGPConfig default --patch \ > '{"spec": {"serviceClusterIPs": [{"cidr": "10.233.0.0/18"}]}}' Successfully patched 1 'BGPConfiguration' resource
Note that the above 10.233.0.0/ 18. The address field is changed to the address field of Service
After the above configuration is completed, you can see the routing information of the synchronized Service address segment in the switch.
Document reference link
https://www.kubernetes.org.cn/9405.html