Binary high availability installation k8s cluster deployment

reference resources Kubernetes full stack Architect (binary high availability installation k8s cluster deployment) - learning notes

1, Binary high availability basic configuration

k8s high availability architecture analysis, high availability Kubernetes cluster planning, setting static ip, please refer to the previous article

1. Configure all node hosts files (send key input to all sessions)
vim /etc/hosts   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6  k8s-master01  k8s-master02  k8s-master03  k8s-node01  k8s-node02
2. CentOS 7 installation yum source is as follows:
curl -o /etc/yum.repos.d/CentOS-Base.repo
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo
sed -i -e '/' -e '/' /etc/yum.repos.d/CentOS-Base.repo
3. Installation of necessary tools
yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git -y
4. Close firewalld, dnsmasq and SELinux on all nodes (NetworkManager needs to be closed for centos7, but not for CentOS8)
systemctl disable --now firewalld 
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager

setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

Check status (must be disabled)


Close swap partition on all nodes, fstab comment swap

swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
5. The synchronization time of all nodes is generally configured for the company's machines

Install ntpdate

rpm -ivh
yum install ntpdate -y

Synchronization time of all nodes. The time synchronization configuration is as follows:

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo 'Asia/Shanghai' >/etc/timezone

Inspection time


Join crontab

crontab -e
# Add the following
*/5 * * * * /usr/sbin/ntpdate
6. Configure limit for all nodes:
ulimit -SHn 65535

vim /etc/security/limits.conf

# Add the following at the end
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited 
7. The Master01 node (cancel sending keys to all sessions) logs in to other nodes without keys. The configuration files and certificates generated during installation are operated on Master01. Cluster management is also operated on Master01. A separate kubectl server is required on Alibaba cloud or AWS. The key configuration is as follows:
ssh-keygen -t rsa

Master01 configure password free login to other nodes

for i in k8s-master01 k8s-master02 k8s-master03 k8s-node01 k8s-node02;do ssh-copy-id -i .ssh/ $i;done
8. All nodes install basic tools (send key input to all sessions)
yum install wget jq psmisc vim net-tools yum-utils device-mapper-persistent-data lvm2 git -y
9. Master01 download installation files (cancel sending key to input all sessions)
cd /root/ ; git clone
10. All nodes (send key to all sessions) upgrade the system and restart. The kernel is not upgraded here. The kernel will be upgraded separately in the next section:
yum update -y --exclude=kernel* && reboot #CentOS7 needs to be upgraded, and CentOS8 can upgrade the system on demand
11. Binary system and kernel upgrade

CentOS7 needs to upgrade the kernel to 4.18 +, and the locally upgraded version is 4.19

Download the kernel on the master01 node (cancel sending key input to all sessions):

cd /root

Transfer from master01 node to other nodes:

for i in k8s-master02 k8s-master03 k8s-node01 k8s-node02;do scp kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm $i:/root/ ; done

Install kernel on all nodes

cd /root && yum localinstall -y kernel-ml*

All nodes change the kernel boot order

grub2-set-default  0 && grub2-mkconfig -o /etc/grub2.cfg

grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"

Check if the default kernel is 4.19

grubby --default-kernel

Restart all nodes and check whether the kernel is 4.19


uname -a
12. Install ipvsadm on all nodes (load balancing):
yum install ipvsadm ipset sysstat conntrack libseccomp -y

All nodes are configured with ipvs module, which is NF in kernel version 4.19 +_ conntrack_ IPv4 has been changed to nf_conntrack, use NF below 4.18_ conntrack_ IPv4:

modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack

vim /etc/modules-load.d/ipvs.conf

	# Add the following

Then execute

systemctl enable --now systemd-modules-load.service

Check whether it is loaded (it can be loaded only after restarting):

lsmod | grep -e ip_vs -e nf_conntrack
13. Enable some necessary kernel parameters in k8s cluster, and configure k8s kernel for all nodes:
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
sysctl --system

net. ipv4. ip_ If forward is not turned on, cross host communication cannot be performed

After all nodes configure the kernel, restart the server to ensure that the kernel is still loaded after restart


lsmod | grep --color=auto -e ip_vs -e nf_conntrack

2, Binary basic component installation

1. Docker installation

Install docker CE 19.03 on all nodes (official recommendation)

yum install docker-ce-19.03.* -y

Since systemd is recommended for the new kubelet, you can change the CgroupDriver of docker to systemd

mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
  "exec-opts": ["native.cgroupdriver=systemd"]

All nodes are set to start Docker automatically:

systemctl daemon-reload && systemctl enable --now docker
2. K8s and etcd installation

(1) Master01 (cancel sending key input to all sessions) Download kubernetes installation package

Visit the official website for the latest version:

Enter the CHANGELOG directory and you can see that the latest version is 1.22. Click Server Binaries to get the download link. If there is an updated version, you need to download the latest version


(2) Download the etcd installation package (3.4.13 is the official recommended version and has been verified)


(3) Unzip the kubernetes installation file. In fact, the binary installation is completed after unzipping

tar -xf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

(4) Extract the etcd installation file

tar -zxvf etcd-v3.4.13-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.4.13-linux-amd64/etcd{,ctl}

(5) Version view

#kubelet --version

Kubernetes v1.22.0-beta.1

# etcdctl version
etcdctl version: 3.4.13
API version: 3.4

(6) Send components to other nodes

MasterNodes='k8s-master02 k8s-master03'
WorkNodes='k8s-node01 k8s-node02'
for NODE in $MasterNodes; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done
for NODE in $WorkNodes; do     scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done

(7) Create / opt/cni/bin directory for all nodes (send key input to all sessions)

mkdir -p /opt/cni/bin

View branch

cd k8s-ha-install/
git branch -a

* master
  remotes/origin/HEAD -> origin/master

Master01 switches to 1.20 X branch (other versions can switch to other branches) (cancel sending key input to all sessions)

git checkout manual-installation-v1.20.x

3, Detailed explanation of binary generated certificate

Binary installation is the most critical step. If one step is wrong, you must pay attention to the correctness of each step

1. Master01 download certificate generation tool
wget "" -O /usr/local/bin/cfssl

wget "" -O /usr/local/bin/cfssljson

chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson
2. Create etcd certificate directory for all Master nodes (send key input to all sessions and cancel node nodes)
mkdir /etc/etcd/ssl -p
3. Create kubernetes related directories for all nodes (send key input to all sessions)
mkdir -p /etc/kubernetes/pki
4. Master01 node generates etcd Certificate (cancel sending key input to all sessions)

(1) CSR file for generating certificate: certificate signature request file, configured with some domain names, companies and companies

# This directory contains the csr files we need to generate the certificate
cd /root/k8s-ha-install/pki

[root@k8s-master01 pki]# ls
admin-csr.json      ca-config.json  etcd-ca-csr.json  front-proxy-ca-csr.json      kubelet-csr.json     manager-csr.json
apiserver-csr.json  ca-csr.json     etcd-csr.json     front-proxy-client-csr.json  kube-proxy-csr.json  scheduler-csr.json

# Generate etcd CA certificate and key of CA certificate
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca

2021/07/23 13:58:44 [INFO] generating a new CA key and certificate from CSR
2021/07/23 13:58:44 [INFO] generate received request
2021/07/23 13:58:44 [INFO] received CSR
2021/07/23 13:58:44 [INFO] generating key: rsa-2048
2021/07/23 13:58:44 [INFO] encoded CSR
2021/07/23 13:58:44 [INFO] signed certificate with serial number 65355458767171380149641516060181865353335743374

(2) View generated key s

ls /etc/etcd/ssl/

etcd-ca.csr  etcd-ca-key.pem  etcd-ca.pem

(3) Issue certificate

cfssl gencert \
   -ca=/etc/etcd/ssl/etcd-ca.pem \
   -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
   -config=ca-config.json \
   -hostname=,k8s-master01,k8s-master02,k8s-master03,,, \
   -profile=kubernetes \
   etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd

(4) View generated certificate

ls /etc/etcd/ssl/

#Generate content
etcd-ca.csr  etcd-ca-key.pem  etcd-ca.pem  etcd.csr  etcd-key.pem  etcd.pem

(5) Copy the certificate to another node

MasterNodes='k8s-master02 k8s-master03'
WorkNodes='k8s-node01 k8s-node02'

for NODE in $MasterNodes; do
     ssh $NODE "mkdir -p /etc/etcd/ssl"
     for FILE in etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem; do
       scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}
5. k8s component certificate

(1) Master01 generate kubernetes certificate

cd /root/k8s-ha-install/pki

cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca

(2) View generated key s

ls /etc/kubernetes/pki

ca.csr  ca-key.pem  ca.pem

(3) Generate client certificate for apiserver

10.96.0. It is the network segment of k8s service. If you need to change the network segment of k8s service, you need to change If it is not a highly available cluster, is the IP of Master01. Here is the highly available vip

cfssl gencert   -ca=/etc/kubernetes/pki/ca.pem   -ca-key=/etc/kubernetes/pki/ca-key.pem   -config=ca-config.json   -hostname=,,,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,,,   -profile=kubernetes   apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

(4) View generated certificates

ls /etc/kubernetes/pki

apiserver.csr  apiserver-key.pem  apiserver.pem  ca.csr  ca-key.pem  ca.pem

(5) Generate the aggregation certificate for apiserver. Requestheader-client-xxx requestheader-allowwd-xxx:aggerator

cfssl gencert   -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca 

cfssl gencert   -ca=/etc/kubernetes/pki/front-proxy-ca.pem   -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem   -config=ca-config.json   -profile=kubernetes   front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client

(6) Generate certificate for controller manage

cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager

# Note that if it is not a highly available cluster, change the address of master 01 to and the port of apiserver to 6443 by default
# Set cluster: set a cluster item
kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server= \

# Set credentials sets a user item
kubectl config set-credentials system:kube-controller-manager \
     --client-certificate=/etc/kubernetes/pki/controller-manager.pem \
     --client-key=/etc/kubernetes/pki/controller-manager-key.pem \
     --embed-certs=true \

# Set an environment item and a context
kubectl config set-context system:kube-controller-manager@kubernetes \
    --cluster=kubernetes \
    --user=system:kube-controller-manager \

# Use an environment as the default
kubectl config use-context system:kube-controller-manager@kubernetes \

(7) Generate the scheduler's certificate

cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler

# Note that if it is not a highly available cluster, change the address of master 01 to and the port of apiserver to 6443 by default
kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server= \

kubectl config set-credentials system:kube-scheduler \
     --client-certificate=/etc/kubernetes/pki/scheduler.pem \
     --client-key=/etc/kubernetes/pki/scheduler-key.pem \
     --embed-certs=true \

kubectl config set-context system:kube-scheduler@kubernetes \
     --cluster=kubernetes \
     --user=system:kube-scheduler \

kubectl config use-context system:kube-scheduler@kubernetes \

(8) Generate admin certificate

cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin

# Note that if it is not a highly available cluster, change the address of master 01 to and the port of apiserver to 6443 by default
kubectl config set-cluster kubernetes     --certificate-authority=/etc/kubernetes/pki/ca.pem     --embed-certs=true     --server=     --kubeconfig=/etc/kubernetes/admin.kubeconfig

kubectl config set-credentials kubernetes-admin     --client-certificate=/etc/kubernetes/pki/admin.pem     --client-key=/etc/kubernetes/pki/admin-key.pem     --embed-certs=true     --kubeconfig=/etc/kubernetes/admin.kubeconfig

kubectl config set-context kubernetes-admin@kubernetes     --cluster=kubernetes     --user=kubernetes-admin     --kubeconfig=/etc/kubernetes/admin.kubeconfig

kubectl config use-context kubernetes-admin@kubernetes     --kubeconfig=/etc/kubernetes/admin.kubeconfig
6. Differentiated certificate

We used the same command to generate admin kubeconfig,scheduler.kubeconfig,controller-manager.kubeconfig, how are they distinguished?

View admin CSR json

cat admin-csr.json

  "CN": "admin", # domain name
  "key": {
    "algo": "rsa",
    "size": 2048
  "names": [
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:masters", # Department, which group does admin belong to
      "OU": "Kubernetes-manual"

The certificate we generated will define a user admin, which belongs to the system:masters group. When k8s is installed, there will be a clusterrole, which is a cluster role, equivalent to a configuration. It has the highest administrative authority of the cluster. At the same time, it will create a clusterrole binding, which will bind admin to the system:masters group, Then all users in this group will have the permissions of this cluster

7. Create serviceaccount key - > Secret

(1) ServiceAccount is k8s an authentication method. When creating a ServiceAccount, a secret bound to it will be created, and this secret will generate a token

openssl genrsa -out /etc/kubernetes/pki/sa.key 2048

openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/

(2) Send certificate to other nodes

for NODE in k8s-master02 k8s-master03; do 
for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do 
scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE};
for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do 
scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE};

(3) View certificate files (23 files in total)

ls /etc/kubernetes/pki/
ls /etc/kubernetes/pki/ |wc -l

[root@k8s-master01 pki]# ls /etc/kubernetes/pki/
admin.csr      apiserver-key.pem  ca.pem                      front-proxy-ca.csr      front-proxy-client-key.pem  scheduler.csr
admin-key.pem  apiserver.pem      controller-manager.csr      front-proxy-ca-key.pem  front-proxy-client.pem      scheduler-key.pem
admin.pem      ca.csr             controller-manager-key.pem  front-proxy-ca.pem      sa.key                      scheduler.pem
apiserver.csr  ca-key.pem         controller-manager.pem      front-proxy-client.csr
[root@k8s-master01 pki]# 
[root@k8s-master01 pki]# ls /etc/kubernetes/pki/ |wc -l

(4) View certificate expiration time (expiration time: 100 years)

cat ca-config.json

  "signing": {
    "default": {
      "expiry": "876000h"
    "profiles": {
      "kubernetes": {
        "usages": [
            "key encipherment",
            "server auth",
            "client auth"
        "expiry": "876000h"

Keywords: Kubernetes

Added by jebster on Sat, 15 Jan 2022 02:48:35 +0200