Preface
In " Packaging and deploying complex applications to the k8s cluster using helm "In this article, we package the application as a chart with helm, which simplifies the deployment process. However, while helm supports Seamless Upgrades for Stateless Applications on the line of Basic Install, replacing the mirrored version to take advantage of the rolling upgrade feature of the K8S RC controller, it has no power for stateful applications.
If you upgrade a MySQL database, it is stateful, so you cannot simply replace the version of the program to complete the database upgrade. There are a series of complex operations to perform: using mysqldump export to import into the new version of the database, or executing scripts to update the database dictionary information using in-place upgrade, which is impossible for helm, so we canComplex logical operations are wrapped in the operator.
Operator provides the following five dimension capabilities, which provide "first day application installation capabilities, also support" the next day "application upgrade maintenance, backup and other life cycle, in-depth analysis, Self Cruise And so on.utilize operator-sdk We can chart helm into operators, use ansible to make operators, or develop operators in go.
Create Operator for Helm Chart
When a helm diagram is made into an operator, it does not have the ability to exceed the helm diagram. In other words, if the helm diagram supports basic installation and seamless upgrade, there will be no more lifecycle features such as backup ~~~ etc. when it is made into an operator, but the operator has some additional capabilities.
First install the SDK client, refer to the documentation Install the Operator SDK CLI.
$ RELEASE_VERSION=v0.18.1 $ curl -LO https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu $ chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu && \ sudo mkdir -p /usr/local/bin/ && \ sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu \ /usr/local/bin/operator-sdk && \ rm operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu $ operator-sdk version operator-sdk version: "v0.18.1"
If the reader follows " Packaging and deploying complex applications to the k8s cluster using helm "This article configures and creates a helm chart and repository, then executes the following command to create an operator for the chart hello:
$ operator-sdk new hello-operator \ --api-version=charts.helm.k8s.io/v1alpha1 \ --kind=Hello --type=helm \ --helm-chart-repo=http://chartmuseum.app.zyl.io \ --helm-chart=hello
Otherwise, you can clone the graph locally and execute the operator-sdk new command to create an operator based on the local directory, as shown below.
$ git clone https://github.com/zylpsrs/helm-example.git $ operator-sdk new hello-operator \ --api-version=charts.helm.k8s.io/v1alpha1 \ --kind=Hello --type=helm \ --helm-chart=helm-example/helm/hello
The above command generates the hello-operator directory with the following structure:
$ tree hello-operator/ hello-operator/ ├── build # Build a mirror directory with Dockerfile files │ └── Dockerfile ├── deploy # operator deployment directory │ ├── crds │ │ ├── hello.helm.k8s.io_hellos_crd.yaml │ │ └── hello.helm.k8s.io_v1alpha1_hello_cr.yaml │ ├── operator.yaml # operator deployment list │ ├── role_binding.yaml # role binding and binding roles to sa │ ├── role.yaml # Role File │ └── service_account.yaml # sa ├── helm-charts # helm charts directory │ └── hello # hello chart └── watches.yaml # What resources do operator s need to monitor
Operator monitors which resources areWatches.yamlThe file defines the parameters that are passed when the operator-sdk new command is executed.For this example, it is in the API:Charts.helmResources of Hello type are monitored on.K8s.io/v1alpha1 and the helm chart executed for this type of request is helm-charts/hello.
--- - group: charts.helm.k8s.io version: v1alpha1 kind: Hello chart: helm-charts/hello
Because we build an operator based on an existing helm diagram, we do not need to adjust the diagram in the helm-charts directory, perform the following command to mirror the operator, then push it to Previous Build a mirror warehouse.As follows:
$ operator-sdk build registry.zyl.io:5000/hello-operator \ --image-builder podman $ podman push registry.zyl.io:5000/hello-operator
After the mirror is built successfully, execute the following command to replace deploy/Operator.yamlREPLACE_IN FILEIMAGE string.
$ cat deploy/operator.yaml ... containers: - name: hello-operator # Replace this with the built image name image: REPLACE_IMAGE ... $ perl -i -ne 's#REPLACE_IMAGE#registry.zyl.io:5000/hello-operator#;print' \ deploy/operator.yaml
Deploy Operator to Cluster
The K8S cluster allows us to register our API with a Custom Resource Definition (CRD). For an operator controller, it listens on a specific API and responds to requests, as defined below for the CRD generated by the SDK for this operator:
$ cat deploy/crds/charts.helm.k8s.io_hellos_crd.yaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition # Use this type to register custom resources metadata: name: hellos.charts.helm.k8s.io # Custom Resource Name spec: group: charts.helm.k8s.io # Custom Resource API Group names: kind: Hello # Custom Resource Type listKind: HelloList plural: hellos singular: hello scope: Namespaced ...
For this example, we chose *.k8s.io as the api group, but it is the k8s reserved group. If an error occurs when registering with the cluster, the solution is as suggested github pr Add a corresponding comment.
# crd has no namespace concept and does not need to specify a namespace through-n <namespace> $ kubectl create -f deploy/crds/charts.helm.k8s.io_hellos_crd.yaml The CustomResourceDefinition "hellos.charts.helm.k8s.io" is invalid: metadata.annotations[api-approved.kubernetes.io]: Required value: protected groups must have approval annotation "api-approved.kubernetes.io", see https://github.com/kubernetes/enhancements/pull/1111 # Edit deploy/crds/Charts.helm.k8s.io_Hellos_Crd.yamlAdd the following comment to the file $ vi deploy/crds/charts.helm.k8s.io_hellos_crd.yaml ... metadata: annotations: "api-approved.kubernetes.io": "https://github.com/kubernetes/kubernetes/pull/78458" ... # Then execute the following command to register: $ kubectl create -f deploy/crds/charts.helm.k8s.io_hellos_crd.yaml customresourcedefinition.apiextensions.k8s.io/hellos.charts.helm.k8s.io created $ kubectl get crd | grep hello hellos.charts.helm.k8s.io 2020-06-19T10:24:13Z
Below we deploy the operator to the demo namespace and execute the command to apply the manifest file in the deploy directory.
$ kubectl -n demo apply -f deploy/ deployment.apps/hello-operator unchanged role.rbac.authorization.k8s.io/hello-operator configured rolebinding.rbac.authorization.k8s.io/hello-operator unchanged serviceaccount/hello-operator unchanged
The above command generates the following objects in the demo namespace:
$ kubectl get pod,svc,deployment NAME READY STATUS RESTARTS AGE pod/hello-operator-756bb58dc5-4g88j 1/1 Running 1 152m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) service/hello-operator-metrics ClusterIP 10.106.213.245 <none> 8383/TCP,8686/TCP NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/hello-operator 1/1 1 1 152m
Request service from Operator via CR
By default, the SDK generates a custom resource (CR) instance request file in the deploy directory that displays default parameters taken from the hello diagramValues.yamlFile, that is, we can configure the same parameters as the hello chart.
$ cat deploy/crds/charts.helm.k8s.io_v1alpha1_hello_cr.yaml apiVersion: charts.helm.k8s.io/v1alpha1 kind: Hello metadata: name: example-hello spec: # Default values copied from <project_dir>/helm-charts/hello/values.yaml affinity: {} autoscaling: ...
After executing the following command and requesting an instance of example through CR, the operator will call the helm chart and respond to this request, the following objects will be visible in the namespace demo:
$ kubectl -n demo apply -f - <<'EOF' apiVersion: charts.helm.k8s.io/v1alpha1 kind: Hello metadata: name: example spec: greeter: replicaCount: 1 EOF $ kubectl -n demo get pod NAME READY STATUS RESTARTS AGE example-greeter-76745b98b8-nswcq 1/1 Running 0 3m28s example-hello-7bc5d74c5b-5tstd 1/1 Running 0 3m28s hello-operator-756bb58dc5-4g88j 1/1 Running 1 173m $ kubectl -n demo get hello NAME AGE example 5m59s
We then execute the following command to update the CR, requesting to configure ingress for the hello application, and checking the operator log reveals the following error with display permissions.
$ kubectl -n demo apply -f - <<'EOF' apiVersion: charts.helm.k8s.io/v1alpha1 kind: Hello metadata: name: example spec: greeter: replicaCount: 1 ingress: enabled: true EOF $ kubectl -n demo logs hello-operator-756bb58dc5-4g88j Unable to continue with install: could not get information about the resource: ingresses.networking.k8s.io "example-hello" is forbidden: User "system:serviceaccount:demo:hello-operator" cannot get resource "ingresses" in API group "networking.k8s.io" in the namespace "demo""
To solve this problem, we can updateRoles.yamlThe file adds the required permissions and then deletes the current pod of the operator. When the new pod starts, it will have the correct permissions to generate ingress, as follows:
$ cat > deploy/role.yaml <<'EOF' - apiGroups: - networking.k8s.io resources: - ingresses verbs: - '*' EOF $ kubectl -n demo apply -f deploy/role.yaml $ kubectl -n demo delete pod hello-operator-756bb58dc5-4g88j $ kubectl -n demo get ingress NAME CLASS HOSTS ADDRESS PORTS AGE example-hello <none> hello.app.zyl.io 192.168.120.6 80 41s
Execute the following command to delete the deploy/ingress object created by the operator and look again after a period of time to see that the object has been rebuilt because the operator monitors the CR object created to ensure consistency of managed back-end objects, such as deploy/ingress, according to the configuration sought by the CR.
$ kubectl -n demo delete deploy,ingress -l app.kubernetes.io/instance=example deployment.apps "example-greeter" deleted deployment.apps "example-hello" deleted ingress.extensions "example-hello" deleted $ kubectl -n demo get deploy,ingress -l app.kubernetes.io/instance=example NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/example-greeter 1/1 1 1 4m30s deployment.apps/example-hello 1/1 1 1 4m30s NAME CLASS HOSTS ADDRESS PORTS AGE ingress.extensions/example-hello <none> hello.app.zyl.io 192.168.120.6 80 4m30s
As above, the operator ensures the number of mirrors defined by the CR as a housekeeper, so we cannot manually adjust the number of mirrors on this backend.
$ kubectl -n demo scale deploy example-greeter --replicas=5 deployment.apps/example-greeter scaled $ kubectl -n demo get deploy example-greeter NAME READY UP-TO-DATE AVAILABLE AGE example-greeter 1/1 1 1 7m23s
Concluding remarks
In this chapter, we make the helm diagram into an operator through operator-sdk, install it into a cluster, then deploy the application through the standard cluster command line tool kubectl without deploying the helm client tools; helm cannot ensure consistency of configuration, when deploying the application through helm, if we adjust the configuration such as deploy, it will result in the existing running configuration and the saved HelmVersions are inconsistent, and operator does solve this problem very well.
In this chapter, we package helm as an operator, but the improvement in its capabilities is not significant. If an operator is implemented through ansible, it supports all the five features illustrated in Chapter 1, which I will explain later.