Packaging Helm diagrams using Operator technology and deploying them to K8S clusters


In " Packaging and deploying complex applications to the k8s cluster using helm "In this article, we package the application as a chart with helm, which simplifies the deployment process. However, while helm supports Seamless Upgrades for Stateless Applications on the line of Basic Install, replacing the mirrored version to take advantage of the rolling upgrade feature of the K8S RC controller, it has no power for stateful applications.

If you upgrade a MySQL database, it is stateful, so you cannot simply replace the version of the program to complete the database upgrade. There are a series of complex operations to perform: using mysqldump export to import into the new version of the database, or executing scripts to update the database dictionary information using in-place upgrade, which is impossible for helm, so we canComplex logical operations are wrapped in the operator.

Operator provides the following five dimension capabilities, which provide "first day application installation capabilities, also support" the next day "application upgrade maintenance, backup and other life cycle, in-depth analysis, Self Cruise And so on.utilize operator-sdk We can chart helm into operators, use ansible to make operators, or develop operators in go.

Create Operator for Helm Chart

When a helm diagram is made into an operator, it does not have the ability to exceed the helm diagram. In other words, if the helm diagram supports basic installation and seamless upgrade, there will be no more lifecycle features such as backup ~~~ etc. when it is made into an operator, but the operator has some additional capabilities.

First install the SDK client, refer to the documentation Install the Operator SDK CLI.

$ curl -LO${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu
$ chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu && \
  sudo mkdir -p /usr/local/bin/ && \
  sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu \
       /usr/local/bin/operator-sdk && \
  rm operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu
$ operator-sdk version
operator-sdk version: "v0.18.1"

If the reader follows " Packaging and deploying complex applications to the k8s cluster using helm "This article configures and creates a helm chart and repository, then executes the following command to create an operator for the chart hello:

$ operator-sdk new hello-operator \ \
  --kind=Hello --type=helm \
  --helm-chart-repo= \

Otherwise, you can clone the graph locally and execute the operator-sdk new command to create an operator based on the local directory, as shown below.

$ git clone
$ operator-sdk new hello-operator \ \
  --kind=Hello --type=helm \

The above command generates the hello-operator directory with the following structure:

$ tree hello-operator/
├── build                    # Build a mirror directory with Dockerfile files
│   └── Dockerfile
├── deploy                   # operator deployment directory
│   ├── crds
│   │   ├── hello.helm.k8s.io_hellos_crd.yaml
│   │   └── hello.helm.k8s.io_v1alpha1_hello_cr.yaml
│   ├── operator.yaml        # operator deployment list
│   ├── role_binding.yaml    # role binding and binding roles to sa
│   ├── role.yaml            # Role File
│   └── service_account.yaml # sa
├── helm-charts              # helm charts directory
│   └── hello                # hello chart
└── watches.yaml             # What resources do operator s need to monitor

Operator monitors which resources areWatches.yamlThe file defines the parameters that are passed when the operator-sdk new command is executed.For this example, it is in the API:Charts.helmResources of Hello type are monitored and the helm chart executed for this type of request is helm-charts/hello.

- group:
  version: v1alpha1
  kind: Hello
  chart: helm-charts/hello

Because we build an operator based on an existing helm diagram, we do not need to adjust the diagram in the helm-charts directory, perform the following command to mirror the operator, then push it to Previous Build a mirror warehouse.As follows:

$ operator-sdk build \
    --image-builder podman
$ podman push

After the mirror is built successfully, execute the following command to replace deploy/Operator.yamlREPLACE_IN FILEIMAGE string.

$ cat deploy/operator.yaml 
        - name: hello-operator
          # Replace this with the built image name
          image: REPLACE_IMAGE
$ perl -i -ne ';print' \

Deploy Operator to Cluster

The K8S cluster allows us to register our API with a Custom Resource Definition (CRD). For an operator controller, it listens on a specific API and responds to requests, as defined below for the CRD generated by the SDK for this operator:

$ cat deploy/crds/charts.helm.k8s.io_hellos_crd.yaml 
kind: CustomResourceDefinition              # Use this type to register custom resources
  name:           # Custom Resource Name
  group:                 # Custom Resource API Group
    kind: Hello                             # Custom Resource Type
    listKind: HelloList
    plural: hellos
    singular: hello
  scope: Namespaced

For this example, we chose * as the api group, but it is the k8s reserved group. If an error occurs when registering with the cluster, the solution is as suggested github pr Add a corresponding comment.

# crd has no namespace concept and does not need to specify a namespace through-n <namespace>
$ kubectl create -f deploy/crds/charts.helm.k8s.io_hellos_crd.yaml 
The CustomResourceDefinition "" is invalid: metadata.annotations[]: Required value: protected groups must have approval annotation "", see

# Edit deploy/crds/Charts.helm.k8s.io_Hellos_Crd.yamlAdd the following comment to the file
$ vi deploy/crds/charts.helm.k8s.io_hellos_crd.yaml
    "": ""

# Then execute the following command to register:
$ kubectl create -f deploy/crds/charts.helm.k8s.io_hellos_crd.yaml created

$ kubectl get crd | grep hello                             2020-06-19T10:24:13Z

Below we deploy the operator to the demo namespace and execute the command to apply the manifest file in the deploy directory.

$ kubectl -n demo apply -f deploy/
deployment.apps/hello-operator unchanged configured unchanged
serviceaccount/hello-operator unchanged

The above command generates the following objects in the demo namespace:

$ kubectl get pod,svc,deployment
NAME                                  READY   STATUS    RESTARTS   AGE
pod/hello-operator-756bb58dc5-4g88j   1/1     Running   1          152m

NAME                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S) 
service/hello-operator-metrics   ClusterIP   <none>        8383/TCP,8686/TCP 

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello-operator   1/1     1            1           152m

Request service from Operator via CR

By default, the SDK generates a custom resource (CR) instance request file in the deploy directory that displays default parameters taken from the hello diagramValues.yamlFile, that is, we can configure the same parameters as the hello chart.

$ cat deploy/crds/charts.helm.k8s.io_v1alpha1_hello_cr.yaml 
kind: Hello
  name: example-hello
  # Default values copied from <project_dir>/helm-charts/hello/values.yaml
  affinity: {}

After executing the following command and requesting an instance of example through CR, the operator will call the helm chart and respond to this request, the following objects will be visible in the namespace demo:

$ kubectl -n demo apply -f - <<'EOF'
kind: Hello
  name: example
    replicaCount: 1

$ kubectl -n demo get pod
NAME                               READY   STATUS    RESTARTS   AGE
example-greeter-76745b98b8-nswcq   1/1     Running   0          3m28s
example-hello-7bc5d74c5b-5tstd     1/1     Running   0          3m28s
hello-operator-756bb58dc5-4g88j    1/1     Running   1          173m

$ kubectl -n demo get hello
example   5m59s

We then execute the following command to update the CR, requesting to configure ingress for the hello application, and checking the operator log reveals the following error with display permissions.

$ kubectl -n demo apply -f - <<'EOF'
kind: Hello
  name: example
    replicaCount: 1
    enabled: true    

$ kubectl -n demo logs hello-operator-756bb58dc5-4g88j 
Unable to continue with install: could not get information about the resource: "example-hello" is forbidden: User "system:serviceaccount:demo:hello-operator" cannot get resource "ingresses" in API group "" in the namespace "demo""

To solve this problem, we can updateRoles.yamlThe file adds the required permissions and then deletes the current pod of the operator. When the new pod starts, it will have the correct permissions to generate ingress, as follows:

$ cat > deploy/role.yaml <<'EOF'
- apiGroups:
  - ingresses
  - '*'
$ kubectl -n demo apply -f deploy/role.yaml
$ kubectl -n demo delete pod hello-operator-756bb58dc5-4g88j
$ kubectl -n demo get ingress
NAME            CLASS    HOSTS              ADDRESS         PORTS   AGE
example-hello   <none>   80      41s

Execute the following command to delete the deploy/ingress object created by the operator and look again after a period of time to see that the object has been rebuilt because the operator monitors the CR object created to ensure consistency of managed back-end objects, such as deploy/ingress, according to the configuration sought by the CR.

$ kubectl -n demo delete deploy,ingress -l
deployment.apps "example-greeter" deleted
deployment.apps "example-hello" deleted
ingress.extensions "example-hello" deleted

$ kubectl -n demo get deploy,ingress -l
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/example-greeter   1/1     1            1           4m30s
deployment.apps/example-hello     1/1     1            1           4m30s

NAME                               CLASS    HOSTS              ADDRESS         PORTS   AGE
ingress.extensions/example-hello   <none>   80      4m30s

As above, the operator ensures the number of mirrors defined by the CR as a housekeeper, so we cannot manually adjust the number of mirrors on this backend.

$ kubectl -n demo scale deploy example-greeter --replicas=5
deployment.apps/example-greeter scaled

$ kubectl -n demo get deploy example-greeter 
example-greeter   1/1     1            1           7m23s

Concluding remarks

In this chapter, we make the helm diagram into an operator through operator-sdk, install it into a cluster, then deploy the application through the standard cluster command line tool kubectl without deploying the helm client tools; helm cannot ensure consistency of configuration, when deploying the application through helm, if we adjust the configuration such as deploy, it will result in the existing running configuration and the saved HelmVersions are inconsistent, and operator does solve this problem very well.

In this chapter, we package helm as an operator, but the improvement in its capabilities is not significant. If an operator is implemented through ansible, it supports all the five features illustrated in Chapter 1, which I will explain later.

Keywords: Linux SDK Kubernetes github Database

Added by kevin_newbie on Fri, 19 Jun 2020 19:26:51 +0300