4.3 collection of common operation and maintenance skills
4.3. 1. Isolation and recovery of nodes
In some cases, we need to isolate some nodes from the scheduling range of K8S. The parameter unscheduled: true can be specified in the spec.
apiVersion: v1
kind: Node
metadata:
name: kubernetes-minion1
lables:
kubernetes.io/hostname: kubernetes-minion1
spec:
unschedulable
Use the kubectl replace command to modify the Node status:
kubectl replace -f unschedule_node.yaml
It should be noted that when a Node is out of the scheduling range, the Pod running on the Node will not stop automatically.
4.3. 2. Node expansion
In the K8S cluster, install Docker, Kubelet and Kube proxy services on the nodes that need to be newly added, specify the Master URL in the startup parameters of Kubelet and Kube proxy as the address of the current K8S cluster Master, and finally start these services. Based on Kubelet's automatic registration mechanism, new nodes will automatically join the existing K8S cluster. After the K8S Master accepts the registration of a new Node, it will automatically bring it into the scheduling range of the current cluster.
4.3. 3 dynamic expansion and scaling of pod
It is implemented through kubectl scale RC < name > -- replica = < number >. If the running Pod is less than number, the Pod will be increased. If the running Pod is greater than number, some running pods will be terminated.
[root@hdss7-21 redis]# kubectl get po NAME READY STATUS RESTARTS AGE redis-slave-qfrk5 1/1 Running 0 97s redis-slave-vbft8 1/1 Running 0 97s
Pod expansion
[root@hdss7-21 redis]# kubectl scale rc redis-slave --replicas=3 replicationcontroller/redis-slave scaled [root@hdss7-21 redis]# kubectl get po NAME READY STATUS RESTARTS AGE redis-slave-lpsgm 0/1 ContainerCreating 0 3s redis-slave-qfrk5 1/1 Running 0 2m49s redis-slave-vbft8 1/1 Running 0 2m49s
pod shrinkage
[root@hdss7-21 redis]# kubectl scale rc redis-slave --replicas=1 replicationcontroller/redis-slave scaled [root@hdss7-21 redis]# kubectl get po NAME READY STATUS RESTARTS AGE redis-slave-lpsgm 1/1 Terminating 0 78s redis-slave-qfrk5 1/1 Terminating 0 4m4s redis-slave-vbft8 1/1 Running 0 4m4s [root@hdss7-21 redis]# kubectl get po NAME READY STATUS RESTARTS AGE redis-slave-vbft8 1/1 Running 0 4m55s
4.3. 4 update the label of the resource object
As an object attribute that can be flexibly defined by the user, the label can still be added, deleted, modified and queried by kubectl label command at any time.
[root@hdss7-21 redis]# kubectl label pod redis-slave-vbft8 role=backend pod/redis-slave-vbft8 labeled
View pod Tags
[root@hdss7-21 redis]# kubectl get po -Lrole NAME READY STATUS RESTARTS AGE ROLE redis-slave-vbft8 1/1 Running 0 16m backend
Delete pod tag
[root@hdss7-21 redis]# kubectl label pod redis-slave-vbft8 role- pod/redis-slave-vbft8 labeled [root@hdss7-21 redis]# kubectl get po -Lrole NAME READY STATUS RESTARTS AGE ROLE redis-slave-vbft8 1/1 Running 0 18m
4.3. 5 schedule the Pod to the specified Node
First, we can kubectl put a specific label on the node:
kubectl label nodes <node-name> <label-key>=<label-value> kubectl label nodes kubernetes-minimon1 zone=north
Then use redis master controller Taking yaml as an example, add nodeSelector definition in the configuration file
apiVersion: v1
kind: ReplicationController
metadata:
name: redis-master
labels:
name: redis-master
spec:
replicas: 2
selector:
name: redis-master
template:
metadata:
labels:
name: redis-master
spec:
containers:
- name: master
image: redis
ports:
- containerPort: 6379
# Look here
nodeSelector:
zone: north
In this way, different node s can be labeled with different labels, such as development environment, test environment and production environment.
Note: if nodeSelector is set in pod, but there is no node with this label in all nodes, even if there are other nodes available, the scheduling will fail.
4.3. 6 application rolling upgrade
K8S provides rolling update to realize rolling upgrade.
Use profile
Use command:
kubectl rolling-update
This command creates a new RC, and then automatically reduces the number of Pod copies in the old RC to 0, while gradually increasing the number of Pod copies in the new RC from 0 to the target value.
In this process, it should be noted that:
- The new RC needs to be in the same Namespace as the old RC.
- The RC name cannot be the same as the old RC name.
- At least one Label in the selector should be different from the old Label to identify it as a new RC.
Direct use command
Use kubectl rolling update plus the -- image parameter to specify the name of the new image to complete the rolling upgrade of Pod.
4.3.7 K8S cluster high availability scheme
It mainly includes two aspects of high availability:
- etcd
- K8S Master assembly
1. etcd high availability scheme
Etcd is the central database in the whole cluster. In order to ensure the high availability of K8S cluster, it is necessary to ensure that the database is not a single point. On the other hand, the data stored by etcd should also consider using reliable storage devices.
2. K8S Master component high availability scheme
In the K8S cluster, the Master service plays the role of the Master control center. The three main services Kube apiserver, Kube controller manager and Kube scheduler maintain the healthy working state of the whole cluster by constantly communicating with Kubelet and Kube proxy on the work Node. If the Master's service cannot access a Node, the Node will be marked as unavailable and the new Pod will not be scheduled. However, the Master itself needs additional monitoring so that the Master does not become a single point of failure of the cluster. Therefore, the Master service also needs to be deployed in a highly available manner.
4.4 resource quota management
At present, the container supports the quota limit of CPU and memory resources.
4.4. 1 specify container quota
apiVersion: v1 kind: ReplicationController metadata: name: redis-master labels: name: redis-master spec: replicas: 1 selector: name: redis-master template: metadata: labels: name: redis-master spec: containers: - name: master image: kubeguide/redis-master ports: - containerPort: 6379 # Resource quota limit resources: limits: cpu: 0.5 memory: 128Mi
The CPU limit calculation method can be explained by referring to the official website, not by simple literal meaning.
If a container exceeds the memory quota during operation, the container may be "killed" and restarted. Therefore, it is necessary to accurately evaluate the memory size required by the container. And beyond the CPU limit without being forcibly "killed".
For more basic knowledge, please check the content of Linux cgroup.
4.2. 2 Global default quota
In addition to adding quota parameters to the specified container in the RC file, you can also define a global default quota template by creating a LimitRange object. This default quota template is loaded back to each Pod and container in the cluster.
The LimitRange object can manage resource quotas at both Pod and Container levels.
apiVersion: v1
kind: LimitRange
metadata:
name: limit-range-1
spec:
limits:
- type: "Pod"
max:
cpu: "2"
memory: 1Gi
min:
cpu: 250m
memory: 32Mi
- type: "Container"
max:
cpu: "2"
memory: 1Gi
min:
cpu: 250m
memory: 32Mi
default:
cpu: 250m
memory: 64Mi
LimitRange is bundled with namespaces. Each namespace can be associated with a different LimitRange as its global default quota configuration. When creating a LimitRange, you can associate it to the specified namespace by specifying -- namespace = namespace on the command line, or you can directly specify the namespace in the definition file:
apiVersion: v1
kind: LimitRange
metadata:
name: limit-range-1
# Specify namespace
namespace: development
spec:
limits:
- type: "Container"
default:
cpu: 250m
memory: 64Mi
4.4. 3 multi tenant quota management
Multi tenancy is embodied by Namespace in K8S. Multi tenancy here can be multiple users, multiple business systems or multiple isolated working environments. The resources in a cluster are always limited. When the cluster is used by multiple tenant applications at the same time, it is necessary to upgrade the resource quota management unit to the tenant level. The purpose can be achieved by loading the corresponding ResourceQuota configuration on the Namespace corresponding to different tenants.
Take the development group and test group as an example, first create the namespace corresponding to the development group:
apiVersion: v1 kind: Namespace metadata: name: devlopment
Then create the ResourceQuota object and specify the Namespace:
apiVersion: v1 kind: ResourceQuota metadata: name: quota-development namespace: devlopment spec: hard: cpu: "32" memory: 256Gi persistentvolumeclaims: "10" pods: "100" replicationcontrollers: "50" resourcequotas: "1" secrets: "20" services: "50"
The operation of the test group is the same as above, except that the namespace needs to be set to test.
After creating a ResourceQuota, you must specify specific resource quota settings for all pods to be created. Otherwise, the creation of Pod fails.
You can then use the command to view the quota usage of a tenant:
# kubectl describe resourcequota <quota name> --namespace=<namespace> kubectl describe resourcequota quota-development --namespace=development
4.5 detailed explanation of k8s network configuration scheme
- 4.5. 1 direct routing
- 4.5. 2 use flannel
- 4.5. 3. Use Open vSwitch
4.6 K8S cluster monitoring
- kube-ui
- cAdvisor
4.7 Trouble Shooting guidance
- View the current runtime information of the K8S object, especially the Event events associated with the object.
- Check the running data of the object and check whether there are parameter errors, association errors, abnormal status and other problems. It may involve the troubleshooting of multiple related objects.
- For service / container problems, you need to check the container and service logs first, and then go deep into the container for troubleshooting.
- For some complex problems, you may need to troubleshoot them in combination with the K8S service log on each node in the cluster. For example, component logs on the Master, Kubelet and Kube proxy service logs on the node.
Summary of frequently asked questions
Pod Pending status
- There are no nodes available for scheduling
- Resource quota management is enabled and no resources are available on the target node of the current Pod
The Pod status is not Ready, and the number of restarts continues to increase
- It is usually because the container's startup command cannot be kept running in the foreground
4.7. Event event of 1 object
# kubectl describe po <pod name>
[root@hdss7-21 redis]# kubectl describe po redis-master-qtczc
Name: redis-master-qtczc
Namespace: default
Priority: 0
Node: hdss7-21.host.com/192.168.50.21
Start Time: Wed, 29 Dec 2021 10:13:44 +0800
Labels: name=redis-master
Annotations: <none>
Status: Running
IP: 172.7.21.7
IPs:
IP: 172.7.21.7
Controlled By: ReplicationController/redis-master
# Omit
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
You can see that there are Event related events, and you can see why some exceptions are caused from Event related events.
4.7. 2 container log
If a Pod contains multiple containers, you need to specify the container name through the - c parameter to view:
kubectl logs <pod name> -c <container name>
4.7.3 K8S system log
If you install on a Linux system and use systemd to manage the K8S service, the journal of systemd will take over the output log of the service program. You can view the log of the system service by using systemd status or journalctl tools.
If you do not use the systemd system to take over the standard output of K8S, you can specify the log storage directory through the startup parameters of some other services:
--logtostderr=false # Do not output to stderr --log-dir=/var/log/kubernetes # Log storage directory --v=0 # glog log level