In the previous article Using MetalLB as LoadBalancer in Kubernetes cluster (Part 1) In, we use MetalLB's Layer2 pattern as the implementation of LoadBalancer to expose the services in Kubernetes cluster outside the cluster.
Remember the IP address pool we assigned to MetalLB in Configmap?
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.1.30-192.168.1.49
The 192.168.1.30-192.168.1.49 IP segment allocated here happens to be in the author's home network. When we use 192.168.1.30, we can successfully access the service.
When I mentioned the disadvantages of Layer2 before, I also missed one point. In addition to the impact on availability and single point network bottleneck during failover, the client needs to be located in the same subnet as the address pool (if the address pool is changed to 192.168.1.30-192.168.1.49, the service will not be accessible). However, in the experimental environment or homelab environment like the author, the first two are not problems, and the latter is a little troublesome in network configuration.
Although the disadvantages are obvious, Layer2 mode has stronger universality. Unlike BGP mode, it needs to support BGP routing. However, these can not stop the author's exploration (strong) cable (compulsive) desire (disease), because there is also an OpenWrt soft route running in My Proxmox virtual machine Yes. This OpenWrt provides external routing services in the form of soft routing through 192.168.1.2, and supports BGP by installing routing software suite.
Before we start, let's take a look at what BPG is and related terms. Students who already know or feel too abstract can skip directly and look back after reading the demo.
What is BGP
BGP is the abbreviation of Border Gateway Protocol.
Border gateway protocol is a core decentralized autonomous routing protocol on the Internet. It realizes the reachability between autonomous systems (AS) by maintaining IP routing table or "prefix" table. It belongs to vector routing protocol. BGP does not use the indicators of traditional internal gateway protocol (IGP), but uses routing based on path, network policy or rule set. Therefore, it is more suitable to be called vector protocol than routing protocol.
The neighbor relationship of BGP (or communication peer / peer entity, peer) is realized through manual configuration. The peer entities establish a session to exchange data through TCP port 179. The BGP router will periodically send 19 bytes of keep alive messages to maintain the connection (the default cycle is 60 seconds). Among various routing protocols, only BGP uses TCP as the transport layer protocol.
BGP running between two or more peer entities in the same AS autonomous system is called iBGP (internal / internal BGP). BGP running between peer entities belonging to different AS is called eBGP (external / external BGP). The router that exchanges information with other AS on the AS boundary is called border/edge router, and the border routers are eBGP opposite ends to each other. In Cisco IOS, the route distance advertised by iBGP is 200, and the priority is lower than that advertised by eBGP and any internal gateway protocol (IGP). In other router implementations, the priority order is that eBGP is higher than IGP, and IGP is higher than iBGP. The main difference between Mar 6, 2022Mar 6, 2022 iBGP and eBGP is the behavior of forwarding routing information. For example, the routing information obtained from eBGP peers will be distributed to all iBGP peers and eBGP peers, but the routing information obtained from iBGP peers will only be distributed to all eBGP peers. All iBGP peers need to be fully interconnected.
Three terms are mentioned here: autonomous system (AS), internal gateway protocol (IGP) and external gateway protocol (EGP).
Autonomous system AS
Let's take a look at the introduction from Wikipedia:
The combination of self-made router and Internet system refers to the combination of self-made router and Internet system. Autonomous system numbers are 16 bit integers, which can be assigned to up to 65536 autonomous systems. The autonomous system number is divided into two ranges. The first range is the public AS N, from 1 to 64511, which can be used on the Internet; The second range is those from 64512 to 65535 called private numbers, which can only be used within an organization's own network.
In short, China Telecom, China Mobile and China Unicom all have their own AS numbers, and there is more than one. If you are interested, you can check the AS numbers in Wikipedia China Internet backbone network Entry.
In addition to the ASN publicly available on the Internet, private numbers can be used internally. For example, I can create several AS using private numbers in my home network.
Internal routing protocol IGP
Quoting from encyclopedia is not the focus of this article, so I won't introduce it too much.
Internal gateway protocol (IGP) refers to a routing protocol used within an autonomous system (AS).
External gateway protocol EGP
External gateway protocol (EGP) is an outdated Internet routing protocol. Replaced by BPG.
Origin of BPG
BPG is created to replace EGP, and can be applied not only outside the AS, but also inside the AS. Therefore, it is divided into EBGP and IBGP.
Having said so much, it may be a little abstract. Go directly to the demo.
Demo
The environment is the same AS before. According to the pre assumption, we want to create two AS: 65000 and 65001< u> The former is the AS where the router and client are located, while the latter is the AS where our cluster service LoadBalancer IP is located < / u >.
We need to make OpenWrt support BGP first.
OpenWrt supports BGP
In order to make OpenWrt support BGP, the routing software suite is used here Quagga . Quagga provides the implementation of OSPFv2, OSPFv3, RIP v1 v2, RIPng and BGP-4.
Quagga architecture consists of core daemon and zebra, which serves as the abstraction layer of the underlying Unix kernel and provides Zserv API to quagga clients through Unix or TCP. It is these Zserv clients that implement the routing protocol and send the routing updates to the zebra daemon. The current implementation of Zserv is:
Quagga's daemon can be configured through a network accessible cli (vty). The CLI follows a similar style to other routing software. An additional tool vtysh is provided, which acts as the aggregation front end of all daemons, allowing all functions of all quagga daemons to be managed in one place.
Execute the following command to complete the installation:
$ opkg update && opkg install quagga quagga-zebra quagga-bgpd quagga-vtysh
After successful installation, it will automatically start and listen to the port:
$ netstat -lantp | grep -e 'zebra\|bgpd' tcp 0 0 0.0.0.0:2601 0.0.0.0:* LISTEN 2984/zebra tcp 0 0 0.0.0.0:2605 0.0.0.0:* LISTEN 3000/bgpd tcp 0 0 :::2601 :::* LISTEN 2984/zebra tcp 0 0 :::2605 :::* LISTEN 3000/bgpd
The 179 port that bpgd listens to for receiving routing information is not seen here, because the route has not been assigned AS yet. Don't worry, let's use the command vtysh to enter vty for configuration:
$ vtysh OpenWrt# conf t OpenWrt(config)# router bgp 65000 OpenWrt(config-router)# neighbor 192.168.1.5 remote-as 65001 OpenWrt(config-router)# neighbor 192.168.1.5 description ubuntu-dev1 OpenWrt(config-router)# neighbor 192.168.1.6 remote-as 65001 OpenWrt(config-router)# neighbor 192.168.1.6 description ubuntu-dev2 OpenWrt(config-router)# exit OpenWrt(config)# exit
Use the show ip bgp summary command in vty to view:
OpenWrt# show ip bgp summary BGP router identifier 192.168.1.2, local AS number 65000 RIB entries 0, using 0 bytes of memory Peers 2, using 18 KiB of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 192.168.1.5 4 65001 0 0 0 0 0 never Active 192.168.1.6 4 65001 0 0 0 0 0 never Active Total number of neighbors 2 Total num. Established sessions 0 Total num. of routes received 0
At this point, we can see that bgpd is already monitoring port 179 by checking port monitoring:
$ netstat -lantp | grep -e 'zebra\|bgpd' tcp 0 0 0.0.0.0:179 0.0.0.0:* LISTEN 3000/bgpd tcp 0 0 0.0.0.0:2601 0.0.0.0:* LISTEN 2984/zebra tcp 0 0 0.0.0.0:2605 0.0.0.0:* LISTEN 3000/bgpd tcp 0 0 :::179 :::* LISTEN 3000/bgpd tcp 0 0 :::2601 :::* LISTEN 2984/zebra tcp 0 0 :::2605 :::* LISTEN 3000/bgpd
After the BGP route is set, it is part of MetalLB.
MetalLB BGP mode
Let's update the configmap:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
peers:
- peer-address: 192.168.1.2
peer-asn: 65000
my-asn: 65001
address-pools:
- name: default
protocol: bgp
addresses:
- 192.168.0.30-192.168.0.49
After the update, you will find that the EXTERNAL-IP of the Service has not been reassigned, and the controller of MetalLB has not been automatically configured. We delete the controller pod to restart:
$ kubectl delete po -n metallb-system -l app=metallb,component=controller pod "controller-66445f859d-vss2t" deleted
You can see that the Service is assigned a new IP address:
$ kubectl get svc -n default NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 25m nginx-lb LoadBalancer 10.43.188.185 192.168.0.30 8080:30381/TCP 21m nginx2-lb LoadBalancer 10.43.208.169 192.168.0.31 8080:32319/TCP 21m
Check the log of speaker POD. You can see that the communication with peer 192.168.1.2 has started and the announcement of IP address has been released:
{"caller":"level.go:63","configmap":"metallb-system/config","event":"peerAdded","level":"info","msg":"peer configured, starting BGP session","peer":"192.168.1.2","ts":"2022-03-06T22:56:17.336335657Z"}
{"caller":"level.go:63","configmap":"metallb-system/config","event":"configLoaded","level":"info","msg":"config (re)loaded","ts":"2022-03-06T22:56:17.336366122Z"}
struct { Version uint8; ASN16 uint16; HoldTime uint16; RouterID uint32; OptsLen uint8 }{Version:0x4, ASN16:0xfde8, HoldTime:0xb4, RouterID:0xc0a80102, OptsLen:0x1e}
{"caller":"level.go:63","event":"sessionUp","level":"info","localASN":65001,"msg":"BGP session established","peer":"192.168.1.2:179","peerASN":65000,"ts":"2022-03-06T22:56:17.337341549Z"}
{"caller":"level.go:63","event":"updatedAdvertisements","ips":["192.168.0.30"],"level":"info","msg":"making advertisements using BGP","numAds":1,"pool":"default","protocol":"bgp","service":"default/nginx-lb","ts":"2022-03-06T22:56:17.341939983Z"}
{"caller":"level.go:63","event":"serviceAnnounced","ips":["192.168.0.30"],"level":"info","msg":"service has IP, announcing","pool":"default","protocol":"bgp","service":"default/nginx-lb","ts":"2022-03-06T22:56:17.341987657Z"}
{"caller":"level.go:63","event":"updatedAdvertisements","ips":["192.168.0.31"],"level":"info","msg":"making advertisements using BGP","numAds":1,"pool":"default","protocol":"bgp","service":"default/nginx2-lb","ts":"2022-03-06T22:56:17.342041554Z"}
{"caller":"level.go:63","event":"serviceAnnounced","ips":["192.168.0.31"],"level":"info","msg":"service has IP, announcing","pool":"default","protocol":"bgp","service":"default/nginx2-lb","ts":"2022-03-06T22:56:17.342056076Z"}
Then you can view the routing table in vty:
OpenWrt# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel, N - NHRP,
> - selected route, * - FIB route
K>* 0.0.0.0/0 via 192.168.1.1, br-lan
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.0.30/32 [20/0] via 192.168.1.5, br-lan, 00:00:06
B>* 192.168.0.31/32 [20/0] via 192.168.1.5, br-lan, 00:00:06
C>* 192.168.1.0/24 is directly connected, br-lan
From the table, we can find the routes of 192.168.0.30/32 and 192.168.0.31/32 bgps.
test
Our new IP access:
$ curl -I 192.168.0.30:8080 HTTP/1.1 200 OK Server: nginx/1.21.6 Date: Sun, 06 Mar 2022 23:10:33 GMT Content-Type: text/html Content-Length: 615 Last-Modified: Tue, 25 Jan 2022 15:03:52 GMT Connection: keep-alive ETag: "61f01158-267" Accept-Ranges: bytes
summary
So far, we have tried two modes of MetalLB: Layer2 has strong universality and does not need any other dependencies, but its disadvantages are also obvious; In addition to relying on the route supporting BGP, BGP mode has no restrictions in other aspects, and there is no availability problem.
BGP should be the ultimate mode of LoadBalancer, but Layer2 is not useless. We still need to make rational choices based on the scenarios used. For example, I will choose Layer2 mode for homelab.
The article is unified in the official account of the cloud.
