WeChat official account: operation and development story, author: Liu Daxian

K3S Description:

K3s (lightweight Kubernetes): similar to RKE, it is also a certified Kubernetes distribution. It is newer than RKE, easier to use and lighter. All components are in a binary file less than 100 MB. From Rancher v2 Starting from 4, the Rancher can be installed on the k3s cluster.

For details, see: https://rancher2.docs.rancher.cn/docs/installation/_index

Rancher brief:

Rancher is a container management platform for companies that use containers. Rancher simplifies the process of using Kubernetes. Developers can Run Kubernetes Everywhere to meet IT requirements and specifications and empower DevOps teams.

For details, see: https://rancher2.docs.rancher.cn/docs/overview/_index

Operating environment:

operating system	host name	IP address	node	to configure
CentOS 7 1810	nginx-master	192.168.111.21	Nginx master server	2C4G
CentOS 7 1810	nginx-backup	192.168.111.22	Nginx standby server	2C4G
ubuntu-18.04.3-live-server	k3s-node1	192.168.111.50	k3s node 1	4C8G
ubuntu-18.04.3-live-server	k3s-node2	192.168.111.51	k3s node 2	4C8G
CentOS 7 1810	k3s-mysql	192.168.111.52	mysql	4C8G

Preparation of system environment before deployment:

Turn off firewall and SeLinux

In order to prevent cluster formation failure due to port problems, we shut down the firewall and selinux in advance

• centos :

  systemctl stop firewalld
  systemctl disable firewalld
  setenforce 0
  sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
  

• Ubuntu:

  sudo ufw disable
  

• Node and Docker function tuning

  https://rancher2.docs.rancher.cn/docs/best-practices/optimize/os/_index
  

Configure the host file:

192.168.111.21 nginx-master
192.168.111.22 nginx-backup
192.168.111.50 k3s-node1
192.168.111.51 k3s-node2
192.168.111.52 k3s-mysql

• Configure the host file and ensure that each machine can communicate with each other through the host name

Tools needed:

The following CLI tools are required for this installation. Make sure these tools are installed and available in $PATH

CLI tools are installed on the k3s node

• kubectl - Kubernetes command line tool

• helm - Kubernetes package management tool.

  See Helm version requirements. Select the version of Helm to install Rancher.

Start deployment:

Install Kubectl:

• Refer to K8S official website for installation. For some special reasons, we use snap here

  sudo apt-get install snapd
  sudo snap install kubectl --classic # The installation here is slow, please wait patiently
  # Verify installation
  kubectl help
  

Install Helm:

• Refer to helm's official website for installation. Helm is the package manager of Kubernetes. The version of helm needs to be higher than v3

  # Download installation package
  wget https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz
  # decompression
  tar zxvf helm-v3.2.1-linux-amd64.tar.gz
  # Move binaries to / usr/local/bin/
  sudo mv linux-amd64/helm /usr/local/bin/helm
  # Verify installation
  helm help
  

To create an Nginx+Keepalived cluster:

This is done on the CentOS node

• Installing Nginx

  # Download the Nginx installation package
  wget http://nginx.org/download/nginx-1.17.10.tar.gz
  # Unzip the installation package
  tar zxvf nginx-1.17.10.tar.gz
  # Install the necessary packages at compile time
  yum install -y gcc gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel libnl3-devel
  # Enter the nginx directory, where we need to use https, and select -- with HTTP during compilation_ ssl_ Module module
  cd nginx-1.17.10
  mkdir -p /usr/local/nginx
  ./configure --prefix=/usr/local/nginx --with-stream
  # Installing nginx
  make && make install
  # Create nginx command soft connection
  ln -s /usr/local/nginx/sbin/nginx /usr/local/bin/nginx
  # Verify installation
  nginx -V
  # start nginx 
  nginx
  

• Install Keepalived

  # Download installation package
  wget https://www.keepalived.org/software/keepalived-2.0.20.tar.gz
  # Unzip the installation package
  tar zxvf keepalived-2.0.20.tar.gz
  # Compile and install keepalived
  cd keepalived-2.0.20
  mkdir /usr/local/keepalived
  ./configure --prefix=/usr/local/keepalived/
  make && make install
  # Configure keepalived as system service
  cp /usr/local/keepalived/sbin/keepalived /usr/sbin/keepalived
  cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived
  touch /etc/init.d/keepalived
  chmod +x /etc/init.d/keepalived # The contents of kept are shown below
  vim /etc/init.d/keepalived
  # Configure keepalived
  mkdir /etc/keepalived/
  cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
  vim /etc/keepalived/keepalived.conf #keepalived. The contents of conf are shown below
  # Start keepalived
  systemctl start keepalived
  systemctl enable keepalived
  # verification
  systemctl status keepalived
  # At this time, keepalived should be running. One is the master and the other is the backup. When the ip addr command is executed on the master, there should be a virtual ip address, and there should be no virtual ip address on the backup
  # visit https://192.168.111.20 Verify configuration
  

  /etc/init.d/keepalived file contents

  #!/bin/sh

  Startup script for the Keepalived daemon

  processname: keepalived

  pidfile: /var/run/keepalived.pid

  config: /etc/keepalived/keepalived.conf

  chkconfig: - 21 79

  description: Start and stop Keepalived

  Source function library

  . /etc/rc.d/init.d/functions

  Source configuration file (we set KEEPALIVED_OPTIONS there)

  . /etc/sysconfig/keepalived

  RETVAL=0

  prog="keepalived"

  start() {
  echo -n $"Starting $prog: "
  daemon keepalived K E E P A L I V E D O P T I O N S R E T V A L = {KEEPALIVED_OPTIONS} RETVAL= KEEPALIVEDO​PTIONSRETVAL=?
  echo
  [ KaTeX parse error: Expected 'EOF', got '&' at position 16: RETVAL -eq 0 ] &̲& touch /var/lo...prog
  }

  stop() {
  echo -n $"Stopping p r o g : " k i l l p r o c k e e p a l i v e d R E T V A L = prog: " killproc keepalived RETVAL= prog:"killprockeepalivedRETVAL=?
  echo
  [ KaTeX parse error: Expected 'EOF', got '&' at position 16: RETVAL -eq 0 ] &̲& rm -f /var/lo...prog
  }

  reload() {
  echo -n $"Reloading p r o g : " k i l l p r o c k e e p a l i v e d − 1 R E T V A L = prog: " killproc keepalived -1 RETVAL= prog:"killprockeepalived−1RETVAL=?
  echo
  }

  See how we were called.

  case " 1 " i n s t a r t ) s t a r t ; ; s t o p ) s t o p ; ; r e l o a d ) r e l o a d ; ; r e s t a r t ) s t o p s t a r t ; ; c o n d r e s t a r t ) i f [ − f / v a r / l o c k / s u b s y s / 1" in start) start ;; stop) stop ;; reload) reload ;; restart) stop start ;; condrestart) if [ -f /var/lock/subsys/ 1"instart)start;;stop)stop;;reload)reload;;restart)stopstart;;condrestart)if[−f/var/lock/subsys/prog ]; then
  stop
  start
  fi
  ;;
  status)
  status keepalived
  RETVAL=$?
  ;;
  *)
  echo "Usage: $0 {start|stop|reload|restart|condrestart|status}"
  RETVAL=1
  esac

  exit $RETVAL

  # /etc/keepalived/keepalived. Contents in conf
  ! Configuration File for keepalived
  
  global_defs {
     router_id 192.168.111.21 # There is only one id in the network, and there should be no duplicate id
  }
  
  vrrp_script chk_nginx {     #Because you want to detect the nginx service status, create a check script
      script "/usr/local/keepalived/check_ng.sh"
      interval 3
  }
  
  vrrp_instance VI_1 {
      state MASTER    # Configure this node as master and BACKUP as standby
      interface ens33    # Set the bound network card
      virtual_router_id 51    # vrrp group: the active and standby vrrp groups should be the same
      priority 120    # Priority, the one with higher priority shall prevail
      advert_int 1    # Inspection interval
      authentication { # authentication
          auth_type PASS
          auth_pass 1111
      }
      virtual_ipaddress { # Virtual IP
          192.168.111.20
      }
      track_script {    # Execute script
          chk_nginx
      }
  }
  

  /usr/local/keepalived/check_ ng. Content in Sh

  #!/bin/bash
  d=date --date today +%Y%m%d_%H:%M:%S
  n=ps -C nginx --no-heading|wc -l
  if [ $n -eq "0" ]; then
  nginx
  n2=ps -C nginx --no-heading|wc -l
  if [ n 2 − e q " 0 " ] ; t h e n e c h o " n2 -eq "0" ]; then echo " n2−eq"0"];thenecho"d nginx down,keepalived will stop" >> /var/log/check_ng.log
  systemctl stop keepalived
  fi
  fi

To install docker Ce:

This is done on the RKE node

# Remove old Docker
sudo apt-get remove docker docker-engine docker.io containerd runc
# Setup Toolkit 
sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
# Add Docker official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
# Add stable apt source
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
# Install docker CE
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
# Verify installation
docker info
# Add the current user to the "docker" user group, and the account added to the user group will be used in the subsequent installation process. The SSH user used for node access must be a member of the docker group on the node
sudo usermod -aG docker $USER

Configure four layer load balancing

Operate in Nginx cluster here

# Update nginx configuration file
# vim /usr/local/nginx/conf/nginx.conf

#user  nobody;
worker_processes  4;
worker_rlimit_nofile 40000;

events {
    worker_connections  8192;
}

stream {
    upstream rancher_servers_http {
        least_conn;
        server 192.168.111.50:80 max_fails=3 fail_timeout=5s;
        server 192.168.111.51:80 max_fails=3 fail_timeout=5s;
    }
    server {
        listen     80;
        proxy_pass rancher_servers_http;
    }

    upstream rancher_servers_https {
        least_conn;
        server 192.168.111.50:443 max_fails=3 fail_timeout=5s;
        server 192.168.111.51:443 max_fails=3 fail_timeout=5s;
    }

    server {
        listen     443;
        proxy_pass rancher_servers_https;
    }
}

Deploying MySQL 5.7

# Download address: https://dev.mysql.com/downloads/mysql/5.7.html#downloads
# Create users and user groups running MySQL database
groupadd -r mysql
useradd -r -g mysql mysql
# Unzip the installation package, change the directory permissions, and create a database directory
tar zxvf mysql-5.7.30-linux-glibc2.12-x86_64.tar.gz
mkdir -p /app/mysql/data
mv mysql-5.7.30-linux-glibc2.12-x86_64/* /app/mysql/
chown -R mysql:mysql /app/mysql
# Initialize database
cd /app/mysql
./bin/mysqld --initialize \
--user=mysql --basedir=/app/mysql/ \
--datadir=/app/mysql/data/
# !! Note the initial password on the last line
7Jlhi:gg?rE0
# Create RSA private key
./bin/mysql_ssl_rsa_setup --datadir=/app/mysql/data/
# Add MySQL to startup and modify / etc / init basedir and datadir in D / mysqld
cp support-files/mysql.server /etc/init.d/mysqld

basedir=/app/mysql
datadir=/app/mysql/data

chkconfig mysqld on
# Modify environment variables
vim /etc/profile
# add to
export PATH=/app/mysql/bin:$PATH
# Make environment variables effective
source /etc/profile

# Backup the / etc / my CNF, create my. In / app/mysql / CNF, and adjust the file attribute to mysql:mysql
mv /etc/my.cnf /etc/my.cnf.bak
touch /app/mysql/my.cnf    # See below for details

# mysql start 
/etc/init.d/mysqld start
# Create mysql Sock soft link
ln -s /app/mysql/mysql.sock /tmp/mysql.sock
# Login with initial password
mysql -uroot -p

# Change the password after successful login
alter user 'root'@'localhost' identified by "12345678";
flush privileges;

# Configure database remote login
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '12345678' WITH GRANT OPTION;
flush privileges;
# Verification strategy

my.cnf

[mysqld]
character-set-server=utf8
datadir=/app/mysql/data
socket=/app/mysql/mysql.sock

Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0

Settings user and group are ignored when systemd is used.

If you need to run mysqld under a different user or group,

customize your systemd unit file for mariadb according to the

instructions in http://fedoraproject.org/wiki/Systemd

include all files from the config directory

!includedir /etc/my.cnf.d

[client]
character-set-server=utf8
socket=/app/mysql/mysql.sock

[mysql]
character-set-server=utf8
socket=/app/mysql/mysql.sock

### Deployment k3s:

Start k3s Server

! Note that this command should be run on all k3s nodes

curl -sfL https://docs.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -s - server
–datastore-endpoint="mysql://root:12345678@tcp(192.168.111.52:3306)/k3s"

verification

sudo k3s kubectl get nodes

When installing K3s on each Rancher Server node, / etc / rancher / K3s / K3s. On the node Create a kubeconfig file in the yaml location. This file contains credentials for full access to the cluster# Copy K3s Yaml to ~ / kube/config

sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

Verify kubectl

sudo kubectl get pods --all-namespaces

kube-system coredns-8655855d6-c26h8 1/1 Running 0 11m
kube-system metrics-server-7566d596c8-v65fd 1/1 Running 0 11m
kube-system helm-install-traefik-ttrfg 0/1 Completed 0 11m
kube-system svclb-traefik-hxmzw 2/2 Running 0 8m16s
kube-system svclb-traefik-zxmg2 2/2 Running 0 8m16s
kube-system traefik-758cd5fc85-xsxbm 1/1 Running 0 8m16s
kube-system local-path-provisioner-6d59f47c7-497rl 1/1 Running 0 11m

### Deploy Rancher:

*   add to Helm Chart Warehouse
    
    ```
    helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
    ```
*   by Rancher establish Namespace
    
    ```
    sudo kubectl create namespace cattle-system
    ```
*   Generate certificate
    
    ```
    mkdir certs
    cd certs
    touch ~/.rnd
    cp /usr/lib/ssl/openssl.cnf ./ # openssl.cnf content has been changed. See below for details
    vim openssl.cnf
    
    openssl genrsa -out cakey.pem 2048
    
    openssl req -x509 -new -nodes -key cakey.pem \
    -days 36500 \
    -out cacerts.pem \
    -extensions v3_ca \
    -subj "/CN=rancher.local.com" \
    -config ./openssl.cnf
    
    openssl genrsa -out server.key 2048
    
    openssl req -new -key server.key \
    -out server.csr \
    -subj "/CN=rancher.local.com" \
    -config ./openssl.cnf
    
    openssl x509 -req -in server.csr \
    -CA cacerts.pem \
    -CAkey cakey.pem \
    -CAcreateserial -out server.crt \
    -days  36500 -extensions v3_req \
    -extfile ./openssl.cnf
    
    openssl x509 -noout -in server.crt -text | grep DNS
    
    cp server.crt tls.crt
    cp server.key tls.key
    ```
*   openssl Modified part
    
    ```
    [req]
    distinguished_name      = req_distinguished_name
    req_extetions           = v3_req
    x509_extensions         = v3_ca
    
    [ v3_req ]
    basicConstraints = CA:FALSE
    keyUsage = nonRepudiation, digitalSignature, keyEncipherment
    extendedKeyUsage = clientAuth, serverAuth
    subjectAltName = @alt_names
    
    [alt_names]
    DNS.1 = rancher.local.com
    
    [ v3_ca ]
    subjectKeyIdentifier=hash
    authorityKeyIdentifier=keyid:always,issuer:always
    basicConstraints = critical,CA:true
    subjectAltName = @alt_names
    ```
*   ca Certificate ciphertext
    
    ```
    sudo kubectl -n cattle-system create secret tls tls-rancher-ingress \
    --cert=./tls.crt --key=./tls.key
    sudo kubectl -n cattle-system create secret generic tls-ca \
    --from-file=cacerts.pem
    ```
*   deploy Rancher colony
    
    ```
     sudo helm install rancher rancher-stable/rancher \
     --namespace cattle-system \
     --set hostname=rancher.local.com \
     --set ingress.tls.source=secret \
     --set privateCA=true
    ```
*   wait for Rancher Cluster operation
    
    ```
    sudo kubectl -n cattle-system rollout status deploy/rancher
    Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
    deployment "rancher" successfully rolled out
    ```
*   If you see the following error:`error: deployment "rancher" exceeded its progress deadline`, You can check by running the following command deployment State of
    
    ```
    sudo kubectl -n cattle-system get deploy rancher
    ```
*   The building is completed, in your hosts File, resolve the domain name to the load balancer, and access `https://rancher.local.com`
    
*   ![picture](https://img-blog.csdnimg.cn/img_convert/d3dbeaea7bd946a9beb54f2f4a5ccd06.png)