Author: adamzhoul, member of OpenYurt

Open readme.com of openYurt MD, after a brief introduction, is Getting started:

 yurtctl convert --provider [minikube|kubeadm|kind] // To convert an existing Kubernetes cluster to an OpenYurt cluster
 yurtctl revert // To uninstall and revert back to the original cluster settings

You can experience OpenYurt with a simple line of command. It feels very convenient.

wait a moment! Why convert/revert instead of install/uninstall?

What does this command do to the cluster?

It seems necessary to find out what it has done before implementing it.

What exactly did yurtctl convert do?

Core process

Following the openYurt source code [1], we sorted out the core process of convert:

1. inspect
   1.1 Check all node Node status is ready
2. Component deployment
  2.1 to node The node is marked with the corresponding label. 
  2.2 use deployment deploy yurt-controller-manager. 
  2.3 use deployment deploy yurt-tunnel-server. 
  2.4 use daemonset deploy yurt-tunnel-agent，Deployed on edge nodes.
  2.5 use deployment deploy yurt-app-manager. 
3. k8s Component modification
  3.1 modify kube-controller-manager.yaml，be used for disable nodelifecycle controller
4. Node transformation
   4.1 write in yurthub.yaml reach/etc/kubernetes/manifests，Start static pod
   4.2 modify kubelet Configure so that kubelet visit yurthub Not direct apiServer

It can be seen that 1 and 2 are nothing special, but conventional service deployment

3. It refers to the operation of the original k8s system components, which requires special attention

4-node transformation does not look complex, but it is very important to the edge.

What does disable nodelicycle controller do?

job content:

1. Query control surface node
2. establish job，adopt` nodeName: {{.nodeName}}` ensure job of pod Schedule to corresponding node Upper execution(adopt nsenter To modify the files on the host). 
3. sed -i 's/--controllers=/--controllers=-nodelifecycle,/g' /etc/kubernetes/manifests/kube-controller-manager.yaml

View Kube controller manager yaml

...
containers:
- command:
- kube-controller-manager
- --allocate-node-cidrs=true
    ...
- --controllers=-nodelifecycle,*,bootstrapsigner,tokencleaner
...

It can be seen that the above series of operations finally modify the startup command of Kube controller manager.

View the Kube controller manager startup Parameter Description:

--controllers Represents the that needs to be turned on controller list

It can be seen that the sed command removes the nodelicycle controller.

So, what does nodelicycle controller do?

In short:

1. Keep listening, kubelet Reported it node information
2. If one node The status is abnormal, or it has not been reported for a long time
  2.1 Expel this node Node or other ---> Cause the above pod Rescheduled

It can be seen that for the edge nodes in the weak network environment, it is easy to hit the abnormal state, resulting in the expulsion of nodes and the rescheduling of pod s.

So I removed it here. Use yurt controller manager instead.

Even if the node heartbeat is lost, the pod in the node in autonomous mode will not be expelled from APIServer.

Note: here, nodes in autonomous mode refer to edge nodes. We usually mark nodes as autonomous nodes by adding annotation.

How is node transformation implemented? What are the differences between cloud nodes and edge nodes?

Similarly, relevant operations are performed in the context of the target host by running a job.

However, compared with the violent use of nsenter, a more elegant way is used here. By attaching the host root path volume to the container.

Modification of kubelet

In the file / var / lib / kubelet / kubedm flags Kubelet in Env_ KUBEADM_ Args add configuration:

--kubeconfig=/var/lib/openyurt/kubelet.conf --bootstrap-kubeconfig=

effect:

1. Parameters:--kubeconfig ， to kubelet Access is specified apiServer Configuration file for.
2. When--kubeconfig File exists,--bootstrap-kubeconfig When empty,
   kubelet You don't need to pass if you start bootstrap-token Replace file certificates and other processes, and read them directly kubeconfig File access apiServer. 
3. because KUBELET_KUBEADM_ARGS yes kubelet Start the last part of the parameters, so it can override the previous parameters.

Where / var / lib / openyurt / kubelet The content of conf is as follows: directly assign the traffic to yurthub:

apiVersion: v1
clusters:
- cluster:
server: http://127.0.0.1:10261
name: default-cluster
contexts:
- context:
cluster: default-cluster
namespace: default
user: default-auth
name: default-context
current-context: default-context
kind: Config
preferences: {}

Startup details of yurthub

The startup parameters of yurthub container are as follows:

command:
- yurthub
- --v=2
- --server-addr=__kubernetes_service_addr__
- --node-name=$(NODE_NAME)
- --join-token=__join_token__
- --working-mode=__working_mode__

We can see from the parameters:

1. server-addr Cloud specified apiServer Address. Note that the address here must be accessible from the public network, otherwise there will be problems under heterogeneous networks.
2. join-token Is to join the node token，Available`kubeadm token create`To create. k8s Provide mechanisms through token Replace the normally accessed kubeconf File.
3. working-mode: cloud/edge. This is the difference between edge nodes and cloud nodes.

We all know that yurthub can be used as cache, which is an important link to solve the problem of edge autonomy. So why does the cloud also need to be deployed? Why distinguish between edge and cloud working modes?

Simply view the yurthub source code CMD / yurthub / APP / start go:

if cfg.WorkingMode == util.WorkingModeEdge {
    cacheMgr, err = cachemanager.NewCacheManager(cfg.StorageWrapper, cfg.SerializerManager, cfg.RESTMapperManager, cfg.SharedFactory)
    ...
} else {
    klog.Infof("%d. disable cache manager for node %s because it is a cloud node", trace, cfg.NodeName)
}
if cfg.WorkingMode == util.WorkingModeEdge {
    ...
    gcMgr, err := gc.NewGCManager(cfg, restConfigMgr, stopCh)
} else {
    klog.Infof("%d. disable gc manager for node %s because it is a cloud node", trace, cfg.NodeName)
}

It can be seen that cloud yurthub does less work on cache and GC.

Check issue[2] to see that the cloud can also use the data filtering capability provided by yurthub to control service traffic

Of course, there is no need to do cache and other work in the cloud.

Command line parameters

During execution, several parameters are important:

--Cloud nodes is used to identify cloud nodes. Multiple nodes are separated by commas: node1,node2

--Deploy yurttunnel tag whether to deploy yurttunnel

--Kubedm conf path marks the path to the kubedm configuration file on the node machine. Default: / etc / SYSTEMd / system / kubelet service. d/10-kubeadm. conf

More parameters can be viewed using yurtctl convert --help.

summary

In short, the convert core does several things:

1. disable k8s of nodelifecontroller，Use your own yurtcontrollermanager To replace its responsibilities.
2. Install your own components, deployment,damenonset Wait for mode deployment. (there is no need to worry about this kind of resource deployment, because the cluster will not be damaged and problems are unlikely to occur.)
3. Edge nodes: starting yurthub static state pod;take kubelet Forward traffic to yurthub. 

It can be seen that the matter of convert is still relatively controllable. Don't worry too much about executing yurtctl convert.

Of course, the last worry should also be completely eliminated by yurtctl revert!

What did yurtctl revert do?

Core process

1. inspect
  1.1 Ensure all node All already ready
2. Delete self deployment components
  2.1 delete yurt-controller-manager deployment And related resources
  2.2 delete yurt-tunnel-agent And related resources
  2.2 delete yurt-tunnel-server And related resources
  2.3 delete yurt-app-manager And related resources
3. k8s Component modification
  3.1 open nodelifecontroller, This is easy to understand, that is, pass the modification command sed Change the order back.
4. Convert cloud and edge nodes to native nodes
  4.1 modify kubelet Configuration, direct connection apiServer
  4.2 delete yurthub Related configurations and directories

The whole process of convert is the reverse operation of convert, which is easy to understand.

It should be noted that. If convert fails, such as job execution timeout or failure. The job will not be deleted.

Even yurtctl revert will not be deleted. The purpose is to keep the site convenient for positioning problems.

If you need to execute yurtctl convert again, you need to delete the job manually.

kubectl get job -n kube-system -A |grep convert
kubectl delete job -n kube-system < job-name>

summary

The yurtctl convert/revert command is one of the quickest ways to experience the openYurt function.

After understanding the implementation principle of these two commands, I know more than half of the technical scheme of openYurt.

Don't worry about executing orders, so easy!

reference material

[1] Source code: https://github.com/openyurtio/openyurt/blob/5063752a9f6645270e3177e39a46df8c23145af2/pkg/yurtctl/cmd/convert/convert.go#L300

[2]issue: https://github.com/openyurtio/openyurt/issues/450