1. Deploy flannel
1.1 Install flannel
kubernetes requires that nodes in the cluster, including master nodes, be interconnected through the Pod segment.flannel uses vxlan technology to create an inter-operable od network for each node, using UDP 8472 as the port.
The first time flanneld starts, it gets the configured PD segment information from etcd, assigns an unused address segment to this node, and then creates the flannedl.1 network interface (or possibly another name, such as flannel1).
Flannel writes the information assigned to its own Pod segment to the/run/flannel/docker file, which docker then uses to set up the docker0 bridge using environment variables to assign IP from this address segment to all Pod containers of this node.
More flannel Reference: 008.Docker Flannel+Etcd Distributed Network Deployment.
Tip: The k8smaster01 node has downloaded the corresponding binary and can be distributed directly to the node.
1.2 Distributing flannel s
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for node_ip in ${NODE_IPS[@]} 4 do 5 echo ">>> ${node_ip}" 6 scp flannel/{flanneld,mk-docker-opts.sh} root@${node_ip}:/opt/k8s/bin/ 7 ssh root@${node_ip} "chmod +x /opt/k8s/bin/*" 8 done
1.3 Create a flannel certificate and key
Tip: The k8smaster01 node has created a CA certificate request file for flanneld and can be distributed directly to the node.
1.4 Distributing certificates and private keys
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for node_ip in ${NODE_IPS[@]} 4 do 5 echo ">>> ${node_ip}" 6 ssh root@${node_ip} "mkdir -p /etc/flanneld/cert" 7 scp flanneld*.pem root@${node_ip}:/etc/flanneld/cert 8 done
1.5 systemd to create flanneld
Tip: The k8smaster01 node has created a systemd that creates a flanneld and can be distributed directly to the node.
1.6 Distribution of flannel systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for node_ip in ${NODE_IPS[@]} 4 do 5 echo ">>> ${node_ip}" 6 scp flanneld.service root@${node_ip}:/etc/systemd/system/ 7 done
2. Start and Verify
2.1 Start flannel
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh 2 [root@k8smaster01 ~]# for node_ip in ${NODE_IPS[@]} 3 do 4 echo ">>> ${node_ip}" 5 ssh root@${node_ip} "systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld" 6 done
2.2 Check flannel startup
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh 2 [root@k8smaster01 ~]# for node_ip in ${NODE_IPS[@]} 3 do 4 echo ">>> ${node_ip}" 5 ssh root@${node_ip} "systemctl status flanneld|grep Active" 6 done
2.3 Check pod segment information
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh 2 [root@k8smaster01 ~]# etcdctl \ 3 --endpoints=${ETCD_ENDPOINTS} \ 4 --ca-file=/etc/kubernetes/cert/ca.pem \ 5 --cert-file=/etc/flanneld/cert/flanneld.pem \ 6 --key-file=/etc/flanneld/cert/flanneld-key.pem \ 7 get ${FLANNEL_ETCD_PREFIX}/config #View Cluster Pod Segment (/16)
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh 2 [root@k8smaster01 ~]# etcdctl \ 3 --endpoints=${ETCD_ENDPOINTS} \ 4 --ca-file=/etc/kubernetes/cert/ca.pem \ 5 --cert-file=/etc/flanneld/cert/flanneld.pem \ 6 --key-file=/etc/flanneld/cert/flanneld-key.pem \ 7 ls ${FLANNEL_ETCD_PREFIX}/subnets #View the list of assigned od subnets (/24) 8 [root@k8smaster01 ~]# etcdctl \ 9 --endpoints=${ETCD_ENDPOINTS} \ 10 --ca-file=/etc/kubernetes/cert/ca.pem \ 11 --cert-file=/etc/flanneld/cert/flanneld.pem \ 12 --key-file=/etc/flanneld/cert/flanneld-key.pem \ 13 get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.8.0-21 #View the node IP and flannel interface addresses corresponding to a Pod segment
Explanation:
172.30.8.0/21 is assigned to node k8snode02 (172.24.8.75);
VtepMAC is the flannel.1 network card MAC address of k8snode02 node.
2.4 Check flannel network information
1 [root@k8snode02 ~]# ip addr show
Interpretation: The address of the flannel.1 network card is the first IP (.0) of the assigned od subnet segment and is/32.
[root@k8smaster01 ~]# ip route show |grep flannel.1
172.30.8.0/21 via 172.30.8.0 dev flannel.1 onlink
172.30.128.0/21 via 172.30.128.0 dev flannel.1 onlink
172.30.208.0/21 via 172.30.208.0 dev flannel.1 onlink
172.30.216.0/21 via 172.30.216.0 dev flannel.1 onlink
Explanation:
Requests to other Pod segments are forwarded to the flannel.1 network card;
flanneld determines which node's interconnected IP the incoming request is sent to based on information from the subnet segment of the etcd, such as ${FLANNEL_ETCD_PREFIX}/subnets/172.30.32.0-21.
2.5 Verify each node flannel
After deploying flannel on each node, check to see if the flannel interface (name may be flannel 0, flannel.0, flannel.1, etc.) has been created:
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh 2 [root@k8smaster01 ~]# for all_ip in ${ALL_IPS[@]} 3 do 4 echo ">>> ${all_ip}" 5 ssh ${all_ip} "/usr/sbin/ip addr show flannel.1|grep -w inet" 6 done
Output:
ping all flannel interface IP on each node to ensure that:
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh 2 [root@k8smaster01 ~]# for all_ip in ${ALL_IPS[@]} 3 do 4 echo ">>> ${all_ip}" 5 ssh ${all_ip} "ping -c 1 172.30.8.0" 6 ssh ${all_ip} "ping -c 1 172.30.32.0" 7 ssh ${all_ip} "ping -c 1 172.30.128.0" 8 ssh ${all_ip} "ping -c 1 172.30.208.0" 9 ssh ${all_ip} "ping -c 1 172.30.216.0" 10 done
