For more information, please pay attention to wl-awesome

• Argo CD
  • brief introduction
  • Core concept
  • Deploy argocd
  • practice
    • Process analysis
    • Source library key files
    • Configuration library key files
    • harbor library configuration information
    • argocd configuration information
    • Sample application
    • Best practices
  • Reference documents

Argo CD

brief introduction

What is Argo CD?

Argo CD is a Kubernetes based declarative GitOps continuous delivery tool

Why use Argo CD

• Applications, configurations, and environments are defined declaratively and are versioned

• Application deployment and lifecycle management are automated, auditable, and easy to understand

working principle

Argo CD follows the GitOps pattern and uses the Git repository as the data source to define the desired state of the application.
Kubernetes application list can be specified in the following ways:

• kustomize application
• helm application
• ksonnet application
• jsonnet
• Directory with yaml|json manifest
• Any custom configuration management tool plug-in

Argo CD can automatically deploy and maintain the desired application state in the specified target environment, which is defined by the manifest file.
The application manifest version can track Git commit to branches, tag s, or fixed to a specific version based on Git commit time.

Argo CD is implemented based on kubernetes controller. It continuously monitors running applications,
And compare the current activity state with the desired target state (as specified in Git repo).
If the active state of a deployed application deviates from the target state, it is considered OutOfSync.
Argo CD visualizes program status differences and provides automatic or manual synchronization tools.

characteristic

• Automatically deploy the application to the specified target environment
• Support a variety of application configuration management tools / templates (kustomize, helm, ksonnet, jsonnet, plain yaml)
• Ability to manage and deploy to multiple k8s clusters
• Single sign on (OIDC, OAuth2, LDAP, SAML 2.0, GitHub, GitLab, Microsoft, LinkedIn)
• Multi tenant and RBAC policies for authorization
• Roll back to the commit specified in the Git repository
• Health analysis of application resources
• Automatically configure drift detection and visualization
• Automatic / manual synchronization is applied to the desired state
• A Web UI that provides a live view of application activity
• CLI for automation and CI integration
• Webhook integration (GitHub, BitBucket, GitLab)
• PRESYNC, sync and postsync hooks to support complex applications (such as blue / green and Canary upgrades)
• Application event audit and tracking API calls
• Prometheus index
• Override ksonnet/helm parameters in Git

Core concept

The following argocd concepts need to have the background of Git, Docker, Kubernetes, Continuous Delivery and GitOps

• Application: a list of Kubernetes resources defined based on Kubernetes CRD
• Application data source type: the type of tool used to build the application (helm, etc.)
• Target status: describes the expected status of the application (such as number of copies, quota, scheduling, etc.), which is described by the application manifest file in the git warehouse
• Activity status: describes the activity status of the application (such as number of copies, quota, scheduling, probe status, etc.)
• Synchronization status: describes the synchronization between the application activity status and the target status (whether it is consistent or not)
• Synchronization: an action to make the application (within the cluster) agree with the target state (git warehouse manifest file description)
• Status of synchronization operation execution: describes whether the synchronization operation is successful
• Refresh: compare the application target status and activity status in git warehouse, and point out the differences
• Health status: describes whether the application is running normally and can provide external services
• Tools: tools for creating application manifest description files (e.g. helm, Kustomize)

What are the items in Argo CD?

The project provides a logical grouping of applications, which is very useful when Argo CD s are used by multiple teams. The project provides the following features:

• Restrict what is deployed (such as Git source code base)
• Restrict the location of application deployment (target k8s cluster and namespace)
• Restrict deployable or undeployable object types (such as RBAC, CRDs, daemon, NetworkPolicy, etc.)
• Define project roles to provide application RBAC (bound to the OIDC group and / or JWT token)

About default items

Each application belongs to a project. If not specified, the application belongs to the default project, which is automatically created,
By default, repo can be deployed from any source to any cluster and all resource types.
The default business group can only be modified, not deleted. When it was initially created, its specification stated as follows:

spec:
  sourceRepos:
  - '*'
  destinations:
  - namespace: '*'
    server: '*'
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'

Deploy argocd

Download statement file

• install.yaml

release

Modify the image reference tag in the file

[root@node1 ~]# grep "image:" install.sh
        image: ghcr.io/dexidp/dex:v2.27.0
        image: quay.io/argoproj/argocd:v2.0.4
        image: redis:6.2.4-alpine
        image: quay.io/argoproj/argocd:v2.0.4
        image: quay.io/argoproj/argocd:v2.0.4
        image: quay.io/argoproj/argocd:v2.0.4

Publish create

kubectl create namespace argocd
kubectl apply -n argocd -f install.yaml

View deployment status

[root@node1 ~]# kubectl get pod -n argocd -w
NAME                                  READY   STATUS    RESTARTS   AGE
argocd-application-controller-0       1/1     Running   0          113s
argocd-dex-server-764699868-28tmj     1/1     Running   0          113s
argocd-redis-675b9bbd9d-dtbzh         1/1     Running   0          113s
argocd-repo-server-59ffd86d98-2w7k4   1/1     Running   0          113s
argocd-server-6d66686c5c-nqfpf        1/1     Running   0          113s

Adjust the service type to NodePort

kubectl -n argocd expose deployments/argocd-server --type="NodePort" --port=8080 --name=argocd-server-nodeport

Get NodePort

[root@node1 ~]# kubectl get service/argocd-server-nodeport -n argocd
NAME                     TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
argocd-server-nodeport   NodePort   10.233.34.101   <none>        8080:31398/TCP   87s

View login password

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d|xargs -n1 echo

Sign in

• Login address: http://NodeIP:31418

practice

An example is given to illustrate how argocd combines gitlab and k8s to realize the cicd process of the application

Other parts of argocd (user management, security audit, custom hook) are not discussed here

Related technologies & Tools:

• gitlab: store source code and Application List
• docker: build image & Container runtime
• harbor: image artifact library, managing images
• jenkins: ci pipeline tool
• k8s: container orchestration tool
• argocd: k8s based cd tool

Process analysis

Code base changes

The developer submits the code and triggers jenkins ci pipeline

ci pipeline execution build

The following steps are included:

• Package build application
• Build application image
• Create an image tag based on the commit id
• Push to image library
• Change configuration library configuration

Change configuration library configuration

The last process of ci pipeline is as follows:

• checkout configuration library
• Use yq tool to change the contents of yaml list file in configuration library (mainly image tag)
• Submit changes to configuration library

cd process

argocd pulls the configuration library list file and compares the contents.

File change - > execute change
File not changed - > continue to observe configuration library changes

Source library key files

Source code engineering

• demo: spring boot based project

Jenkinsfile content

pipeline {

    agent any

    environment {
        DEMO_IMAGE_TAG="harbor.wl.com/library/demo"
        DOCKER_REGISTRY_DOMAIN="harbor.wl.com"
        DOCKER_CREDENTIAL_ID = 'harbor-secret'
        GIT_CREDENTIAL_ID='d145edf3-929c-4efa-aa46-48ea0cf4336e'
        GIT_CONFIG_REPO_URL="192.168.1.1:80/demo-group/demo.git"
    }

    stages {

        stage ('checkout scm') {
            steps {
                checkout(scm)
            }
        }
        
        // Get the commit id submitted by git
        stage('get commit id...') {
            steps {
                script {
                    env.GIT_COMMIT_ID = sh (script: 'git rev-parse --short HEAD', returnStdout: true).trim()
                    }   
            }
        }

        // Build the demo application image based on the Dockerfile content and generate two versions of tag: Latest & & commit ID
        stage ('build demo image...') {
            steps {
                sh '''
                    sudo docker build -t $DEMO_IMAGE_TAG -f Dockerfile .
                    sudo docker tag $DEMO_IMAGE_TAG $DEMO_IMAGE_TAG:$GIT_COMMIT_ID
                '''
            }
        }

        // Push the image to the local Harbor library, and the bill is managed by jenkins
        stage ('publish image with portal...') {
            steps {
                withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$DOCKER_CREDENTIAL_ID" ,)]) {
                        sh 'sudo echo "$DOCKER_PASSWORD" | sudo docker login $DOCKER_REGISTRY_DOMAIN -u "$DOCKER_USERNAME" --password-stdin'
                        sh '''
                        sudo docker push "$DEMO_IMAGE_TAG"
                        sudo docker push "$DEMO_IMAGE_TAG:$GIT_COMMIT_ID"
                        '''
                }
            }
        }

        // checkout configuration library
        stage ('checkout config repo ...') {
            steps {
                checkout([$class: 'GitSCM', branches: [[name: '*/master']], extensions: [], userRemoteConfigs: [[credentialsId: "$GIT_CREDENTIAL_ID", url: "http://${GIT_CONFIG_REPO_URL}"]]])
            }
        }

        // Change demo / demo. In the demo config library Yaml file internal image tag
        // Commit changes to the demo config library
        stage ('commit config repo changes ...') {
            steps {
                withCredentials([usernamePassword(credentialsId: "$GIT_CREDENTIAL_ID", passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) {
                        sh '''
                        echo "#$GIT_COMMIT_ID#"
                        tag=$DEMO_IMAGE_TAG:$GIT_COMMIT_ID
                        tag=$tag yq eval ".spec.template.spec.containers[0].image = strenv(tag)" -i demo/demo.yaml
                        git add demo/demo.yaml
                        git commit -m "modify version"
                        git config --global push.default simple
                        git push http://$GIT_USERNAME:$GIT_PASSWORD@${GIT_CONFIG_REPO_URL} HEAD:master 
                        '''
                }
            }
        }
    }
}

Dockerfile content

FROM harbor.wl.com/library/maven:3.8.1 AS builder
WORKDIR /usr/local
ADD . .
RUN mvn clean package

FROM harbor.wl.com/library/openjdk-1.8:alpine
COPY --from=builder /usr/local/demo/target/demo-0.0.1-SNAPSHOT.jar /opt/app.jar
EXPOSE 8080

Configuration library key files

Demo config configuration library level and list content

Hierarchy

demo-config
└── demo
    ├── demo-svc.yaml
    └── demo.yaml

demo.yaml content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  labels:
    app: demo-app
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      containers:
        - name: demo-app
          image: harbor.wl.com/library/demo:da28fcb
          imagePullPolicy: Always
          args:
            - java
            - '-Xms2048m'
            - '-Xmx2048m'
            - '-jar'
            - /opt/app.jar
            - '--server.port=8080'
            - '--spring.profiles.active=dev'
          livenessProbe:
            failureThreshold: 10
            httpGet:
              path: /actuator/health
              port: 7002
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 10
            httpGet:
              path: /actuator/health
              port: 7002
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          ports:
            - containerPort: 8080
              name: http-8080
              protocol: TCP
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30

demo-svc.yaml content

---
apiVersion: v1
kind: Service
metadata:
  name: demo-svc
  labels:
    app: demo-svc
spec:
  ports:
    - name: http-8080
      port: 80
      protocol: TCP
      targetPort: 8080
  selector:
    app: demo-app
  sessionAffinity: None
  type: ClusterIP

harbor library configuration information

Configure the image cleaning policy to avoid too many garbage images

argocd configuration information

Configure warehouse

The web console enters the warehouse configuration interface

Click CONNECT REPO USING HTTPS to add the warehouse

Click CONNECT for configuration related information

View warehouse status under Project

Configure cluster

Click Settings - > cluster

Edit the cluster information, and the namespace value is empty (it will be automatically filled as All namespaces after saving)

Create project (logical grouping)

Click Settings - > project

Create demo project

Configure git warehouse and k8s cluster information associated with the project

Create application

New application

Configure application

This completes the process configuration

Sample application

The following shows the cd application of the actual development project

Apply associated resource objects

Application synchronization information

Best practices

Separation of configuration library and source code library

Use a separate Git repository to save the kubernetes list and separate the configuration from the application source code. It is highly recommended for the following reasons:

• Clear separation of application code and application configuration. Sometimes you want to modify only the manifest without triggering the entire CI build.
  For example, if you just want to increase the number of copies in the deployment specification, you may not want to trigger the build (because the build cycle may be long)
• Cleaner audit logs. For audit purposes, only the historical change records of the configuration library are saved, rather than mixed with the log records of daily development submissions.
• In the microservice scenario, an application may be composed of services built by multiple Git libraries, but deployed as a single unit (such as in the same pod).
  Generally, microservice applications are composed of services with different versions and different release cycles (such as ELK, Kafka + Zookeeper).
  Storing the configuration manifest in a single source code base for a single component may not make sense
• Separation of access. The developer who develops the application is not necessarily the same person who can / should be pushed to the production environment, whether intentionally or unintentionally.
  By using a separate library, you can grant commit access to the source code library instead of the application configuration library
• In the automated CI Pipeline scenario, pushing inventory changes to the same Git repository may trigger an infinite loop of build jobs and Git commit triggers.
  This can be prevented by using a separate repo to push configuration changes.

Make sure that the manifest in the Git version is truly immutable

When using template tools such as helm or kustomize, the contents of the list may change over time.
This is usually caused by changes to the upstream helm library or kustomize library.

Take the following kustomization Yaml as an example

bases:
- github.com/argoproj/argo-cd//manifests/cluster-install

Since this is not a stable goal, the list of this custom application may change suddenly without even making any changes to its Git repository. (such as git master branch)

A better choice is to use Git tags or submit the version of SHA. For example:

bases:
- github.com/argoproj/argo-cd//manifests/cluster-install?ref=v0.11.1

Reference documents

argocd document