Building a highly available Rancher cluster based on K3S

WeChat official account: operation and development story, author: Liu Daxian

K3S Description:

K3s (lightweight Kubernetes): similar to RKE, it is also a certified Kubernetes distribution. It is newer than RKE, easier to use and lighter. All components are in a binary file less than 100 MB. From Rancher v2 Starting from 4, the Rancher can be installed on the k3s cluster.

For details, see:

Rancher brief:

Rancher is a container management platform for companies that use containers. Rancher simplifies the process of using Kubernetes. Developers can Run Kubernetes Everywhere to meet IT requirements and specifications and empower DevOps teams.

For details, see:

Operating environment:

operating systemhost nameIP addressnodeto configure
CentOS 7 1810nginx-master192.168.111.21Nginx master server2C4G
CentOS 7 1810nginx-backup192.168.111.22Nginx standby server2C4G
ubuntu-18.04.3-live-serverk3s-node1192.168.111.50k3s node 14C8G
ubuntu-18.04.3-live-serverk3s-node2192.168.111.51k3s node 24C8G
CentOS 7 1810k3s-mysql192.168.111.52mysql4C8G

Preparation of system environment before deployment:

Turn off firewall and SeLinux

In order to prevent cluster formation failure due to port problems, we shut down the firewall and selinux in advance

  • centos :

    systemctl stop firewalld
    systemctl disable firewalld
    setenforce 0
    sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
  • Ubuntu:

    sudo ufw disable
  • Node and Docker function tuning

Configure the host file: nginx-master nginx-backup k3s-node1 k3s-node2 k3s-mysql
  • Configure the host file and ensure that each machine can communicate with each other through the host name

Tools needed:

The following CLI tools are required for this installation. Make sure these tools are installed and available in $PATH

CLI tools are installed on the k3s node

  • kubectl - Kubernetes command line tool

  • helm - Kubernetes package management tool.

    See Helm version requirements. Select the version of Helm to install Rancher.

Start deployment:

Install Kubectl:

  • Refer to K8S official website for installation. For some special reasons, we use snap here

    sudo apt-get install snapd
    sudo snap install kubectl --classic # The installation here is slow, please wait patiently
    # Verify installation
    kubectl help

Install Helm:

  • Refer to helm's official website for installation. Helm is the package manager of Kubernetes. The version of helm needs to be higher than v3

    # Download installation package
    # decompression
    tar zxvf helm-v3.2.1-linux-amd64.tar.gz
    # Move binaries to / usr/local/bin/
    sudo mv linux-amd64/helm /usr/local/bin/helm
    # Verify installation
    helm help

To create an Nginx+Keepalived cluster:

This is done on the CentOS node

  • Installing Nginx

    # Download the Nginx installation package
    # Unzip the installation package
    tar zxvf nginx-1.17.10.tar.gz
    # Install the necessary packages at compile time
    yum install -y gcc gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel libnl3-devel
    # Enter the nginx directory, where we need to use https, and select -- with HTTP during compilation_ ssl_ Module module
    cd nginx-1.17.10
    mkdir -p /usr/local/nginx
    ./configure --prefix=/usr/local/nginx --with-stream
    # Installing nginx
    make && make install
    # Create nginx command soft connection
    ln -s /usr/local/nginx/sbin/nginx /usr/local/bin/nginx
    # Verify installation
    nginx -V
    # start nginx 
  • Install Keepalived

    # Download installation package
    # Unzip the installation package
    tar zxvf keepalived-2.0.20.tar.gz
    # Compile and install keepalived
    cd keepalived-2.0.20
    mkdir /usr/local/keepalived
    ./configure --prefix=/usr/local/keepalived/
    make && make install
    # Configure keepalived as system service
    cp /usr/local/keepalived/sbin/keepalived /usr/sbin/keepalived
    cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived
    touch /etc/init.d/keepalived
    chmod +x /etc/init.d/keepalived # The contents of kept are shown below
    vim /etc/init.d/keepalived
    # Configure keepalived
    mkdir /etc/keepalived/
    cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
    vim /etc/keepalived/keepalived.conf #keepalived. The contents of conf are shown below
    # Start keepalived
    systemctl start keepalived
    systemctl enable keepalived
    # verification
    systemctl status keepalived
    # At this time, keepalived should be running. One is the master and the other is the backup. When the ip addr command is executed on the master, there should be a virtual ip address, and there should be no virtual ip address on the backup
    # visit Verify configuration

    /etc/init.d/keepalived file contents


    Startup script for the Keepalived daemon

    processname: keepalived

    pidfile: /var/run/

    config: /etc/keepalived/keepalived.conf

    chkconfig: - 21 79

    description: Start and stop Keepalived

    Source function library

    . /etc/rc.d/init.d/functions

    Source configuration file (we set KEEPALIVED_OPTIONS there)

    . /etc/sysconfig/keepalived



    start() {
    echo -n $"Starting $prog: "
    [ KaTeX parse error: Expected 'EOF', got '&' at position 16: RETVAL -eq 0 ] &̲& touch /var/lo...prog

    stop() {
    echo -n $"Stopping p r o g : " k i l l p r o c k e e p a l i v e d R E T V A L = prog: " killproc keepalived RETVAL= prog:"killprockeepalivedRETVAL=?
    [ KaTeX parse error: Expected 'EOF', got '&' at position 16: RETVAL -eq 0 ] &̲& rm -f /var/lo...prog

    reload() {
    echo -n $"Reloading p r o g : " k i l l p r o c k e e p a l i v e d − 1 R E T V A L = prog: " killproc keepalived -1 RETVAL= prog:"killprockeepalived−1RETVAL=?

    See how we were called.

    case " 1 " i n s t a r t ) s t a r t ; ; s t o p ) s t o p ; ; r e l o a d ) r e l o a d ; ; r e s t a r t ) s t o p s t a r t ; ; c o n d r e s t a r t ) i f [ − f / v a r / l o c k / s u b s y s / 1" in start) start ;; stop) stop ;; reload) reload ;; restart) stop start ;; condrestart) if [ -f /var/lock/subsys/ 1"instart)start;;stop)stop;;reload)reload;;restart)stopstart;;condrestart)if[−f/var/lock/subsys/prog ]; then
    status keepalived
    echo "Usage: $0 {start|stop|reload|restart|condrestart|status}"

    exit $RETVAL

    # /etc/keepalived/keepalived. Contents in conf
    ! Configuration File for keepalived
    global_defs {
       router_id # There is only one id in the network, and there should be no duplicate id
    vrrp_script chk_nginx {     #Because you want to detect the nginx service status, create a check script
        script "/usr/local/keepalived/"
        interval 3
    vrrp_instance VI_1 {
        state MASTER    # Configure this node as master and BACKUP as standby
        interface ens33    # Set the bound network card
        virtual_router_id 51    # vrrp group: the active and standby vrrp groups should be the same
        priority 120    # Priority, the one with higher priority shall prevail
        advert_int 1    # Inspection interval
        authentication { # authentication
            auth_type PASS
            auth_pass 1111
        virtual_ipaddress { # Virtual IP
        track_script {    # Execute script

    /usr/local/keepalived/check_ ng. Content in Sh

    d=date --date today +%Y%m%d_%H:%M:%S
    n=ps -C nginx --no-heading|wc -l
    if [ $n -eq "0" ]; then
    n2=ps -C nginx --no-heading|wc -l
    if [ n 2 − e q " 0 " ] ; t h e n e c h o " n2 -eq "0" ]; then echo " n2−eq"0"];thenecho"d nginx down,keepalived will stop" >> /var/log/check_ng.log
    systemctl stop keepalived

To install docker Ce:

This is done on the RKE node

# Remove old Docker
sudo apt-get remove docker docker-engine containerd runc
# Setup Toolkit 
sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
# Add Docker official GPG key
curl -fsSL | sudo apt-key add -
# Add stable apt source
sudo add-apt-repository \
   "deb [arch=amd64] \
   $(lsb_release -cs) \
# Install docker CE
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli
# Verify installation
docker info
# Add the current user to the "docker" user group, and the account added to the user group will be used in the subsequent installation process. The SSH user used for node access must be a member of the docker group on the node
sudo usermod -aG docker $USER

Configure four layer load balancing

Operate in Nginx cluster here

# Update nginx configuration file
# vim /usr/local/nginx/conf/nginx.conf

#user  nobody;
worker_processes  4;
worker_rlimit_nofile 40000;

events {
    worker_connections  8192;

stream {
    upstream rancher_servers_http {
        server max_fails=3 fail_timeout=5s;
        server max_fails=3 fail_timeout=5s;
    server {
        listen     80;
        proxy_pass rancher_servers_http;

    upstream rancher_servers_https {
        server max_fails=3 fail_timeout=5s;
        server max_fails=3 fail_timeout=5s;

    server {
        listen     443;
        proxy_pass rancher_servers_https;

Deploying MySQL 5.7

# Download address:
# Create users and user groups running MySQL database
groupadd -r mysql
useradd -r -g mysql mysql
# Unzip the installation package, change the directory permissions, and create a database directory
tar zxvf mysql-5.7.30-linux-glibc2.12-x86_64.tar.gz
mkdir -p /app/mysql/data
mv mysql-5.7.30-linux-glibc2.12-x86_64/* /app/mysql/
chown -R mysql:mysql /app/mysql
# Initialize database
cd /app/mysql
./bin/mysqld --initialize \
--user=mysql --basedir=/app/mysql/ \
# !! Note the initial password on the last line
# Create RSA private key
./bin/mysql_ssl_rsa_setup --datadir=/app/mysql/data/
# Add MySQL to startup and modify / etc / init basedir and datadir in D / mysqld
cp support-files/mysql.server /etc/init.d/mysqld


chkconfig mysqld on
# Modify environment variables
vim /etc/profile
# add to
export PATH=/app/mysql/bin:$PATH
# Make environment variables effective
source /etc/profile

# Backup the / etc / my CNF, create my. In / app/mysql / CNF, and adjust the file attribute to mysql:mysql
mv /etc/my.cnf /etc/my.cnf.bak
touch /app/mysql/my.cnf    # See below for details

# mysql start 
/etc/init.d/mysqld start
# Create mysql Sock soft link
ln -s /app/mysql/mysql.sock /tmp/mysql.sock
# Login with initial password
mysql -uroot -p

# Change the password after successful login
alter user 'root'@'localhost' identified by "12345678";
flush privileges;

# Configure database remote login
flush privileges;
# Verification strategy



Disabling symbolic-links is recommended to prevent assorted security risks


Settings user and group are ignored when systemd is used.

If you need to run mysqld under a different user or group,

customize your systemd unit file for mariadb according to the

instructions in

include all files from the config directory

!includedir /etc/my.cnf.d



### Deployment k3s:

Start k3s Server

! Note that this command should be run on all k3s nodes

curl -sfL | INSTALL_K3S_MIRROR=cn sh -s - server


sudo k3s kubectl get nodes

When installing K3s on each Rancher Server node, / etc / rancher / K3s / K3s. On the node Create a kubeconfig file in the yaml location. This file contains credentials for full access to the cluster# Copy K3s Yaml to ~ / kube/config

sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

Verify kubectl

sudo kubectl get pods --all-namespaces

kube-system coredns-8655855d6-c26h8 1/1 Running 0 11m
kube-system metrics-server-7566d596c8-v65fd 1/1 Running 0 11m
kube-system helm-install-traefik-ttrfg 0/1 Completed 0 11m
kube-system svclb-traefik-hxmzw 2/2 Running 0 8m16s
kube-system svclb-traefik-zxmg2 2/2 Running 0 8m16s
kube-system traefik-758cd5fc85-xsxbm 1/1 Running 0 8m16s
kube-system local-path-provisioner-6d59f47c7-497rl 1/1 Running 0 11m

### Deploy Rancher:

*   add to Helm Chart Warehouse
    helm repo add rancher-stable
*   by Rancher establish Namespace
    sudo kubectl create namespace cattle-system
*   Generate certificate
    mkdir certs
    cd certs
    touch ~/.rnd
    cp /usr/lib/ssl/openssl.cnf ./ # openssl.cnf content has been changed. See below for details
    vim openssl.cnf
    openssl genrsa -out cakey.pem 2048
    openssl req -x509 -new -nodes -key cakey.pem \
    -days 36500 \
    -out cacerts.pem \
    -extensions v3_ca \
    -subj "/" \
    -config ./openssl.cnf
    openssl genrsa -out server.key 2048
    openssl req -new -key server.key \
    -out server.csr \
    -subj "/" \
    -config ./openssl.cnf
    openssl x509 -req -in server.csr \
    -CA cacerts.pem \
    -CAkey cakey.pem \
    -CAcreateserial -out server.crt \
    -days  36500 -extensions v3_req \
    -extfile ./openssl.cnf
    openssl x509 -noout -in server.crt -text | grep DNS
    cp server.crt tls.crt
    cp server.key tls.key
*   openssl Modified part
    distinguished_name      = req_distinguished_name
    req_extetions           = v3_req
    x509_extensions         = v3_ca
    [ v3_req ]
    basicConstraints = CA:FALSE
    keyUsage = nonRepudiation, digitalSignature, keyEncipherment
    extendedKeyUsage = clientAuth, serverAuth
    subjectAltName = @alt_names
    DNS.1 =
    [ v3_ca ]
    basicConstraints = critical,CA:true
    subjectAltName = @alt_names
*   ca Certificate ciphertext
    sudo kubectl -n cattle-system create secret tls tls-rancher-ingress \
    --cert=./tls.crt --key=./tls.key
    sudo kubectl -n cattle-system create secret generic tls-ca \
*   deploy Rancher colony
     sudo helm install rancher rancher-stable/rancher \
     --namespace cattle-system \
     --set \
     --set ingress.tls.source=secret \
     --set privateCA=true
*   wait for Rancher Cluster operation
    sudo kubectl -n cattle-system rollout status deploy/rancher
    Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
    deployment "rancher" successfully rolled out
*   If you see the following error:`error: deployment "rancher" exceeded its progress deadline`, You can check by running the following command deployment State of
    sudo kubectl -n cattle-system get deploy rancher
*   The building is completed, in your hosts File, resolve the domain name to the load balancer, and access ``
*   ![picture](

Keywords: Operation & Maintenance Docker Kubernetes

Added by wiseass on Mon, 03 Jan 2022 23:32:20 +0200