Practice of CD Process Based on Gitlab

> DevOps (the combination of English Development and Operations) is a collective term for a set of processes, methods and systems used to promote communication, collaboration and integration among development (application/software engineering), technology operations and quality assurance (QA) departments. Its emergence is due to the increasingly clear recognition in the software industry that in order to deliver software products and services on time, development and operation must work closely together. > > Baidu Encyclopedia

DevOps has been putting forward for some years. This article does not involve related concepts. It only takes Gitlab-based CD (Continuous Delivery) process as an example to illustrate the composition, configuration and use of tools in practice.

When it comes to CI (Continuous Integration), you have to mention Jenkins, but this process does not use it for the following reasons:

Gitlab itself has CI functionality, better integration and simpler
Jenkins is too complex for DevOps, or unfriendly to developers. Testing deployment of a project often requires the intervention of operations personnel, while Gitlab CI process developers can control it by themselves
Jenkins does not have natural pipeline functionality and needs to be implemented through blueocean plug-ins

> Note > > With a Spring > Boot Project (Java Project) is an example of how to do source control, code quality check, compilation and deployment. > > The environment IP of this article is 127.0.0.1, please modify it yourself.

Version Control Using Gitlab
After the code Push:
1. Call SonarQube for quality checks
2. Call Gitlab CI for compilation and unit testing (CI)
3. Call Gitlab CI for deployment (CD)

Environmental preparation

Install Docker.

curl -sSL https://get.docker.com/ | sh

Install Gitlab By Docker.

docker run -d  \
   -p 1443:443 -p 180:80 -p 122:22 \  #Open ports. Note that 180 Web ports are open here.
   --name gitlab \
   --env GITLAB_OMNIBUS_CONFIG="external_url 'http://127.0.0.1:180'; nginx ['listen_port'] = 80; "\ Web URL, please replace `127.0.0.1''with the real host IP, which corresponds to the address of GIT, such as `http://root@127.0.1:180/root/test.git', with particular attention to the need to specify that the Nginx port is `80' (the real port of the container), otherwise the Nginx port will be changed to `180', which will result in inaccessibility.
   --restart always \
   -v /data/gitlab/config:/etc/gitlab \  # Mapping configuration file directory
   -v /data/gitlab/logs:/var/log/gitlab \
   -v /data/gitlab/data:/var/opt/gitlab \  # Mapped Data File Directory
   gitlab/gitlab-ce:latest

Install Gitlab-runner.

Refer to https://docs.gitlab.com/runner/install/linux-repository.html

> Tip > > Gitlab > CE has its own CI function, but only one with scheduling and monitoring functions. The actual execution capability needs to be provided by Gitlab-runner. The advantage is that Gitlab will not affect performance because of CI execution, while using independent Gitlab-runner can deploy on different hosts more freely.

> Tip > > Because this article uses Executor of docker type, it is more convenient to install it directly on the host.

Modify permissions.

The default gitlab-runner creates a gitlab-runner user. Add this user to the sudo group (password-free): gitlab-runner ALL=(ALL) NOPASSWD: ALL

This document requires two runner s:

[root[@host1](https://My.oschina.net/host1) target] gitlab-ci-multi-runner register registration `runner`
Running in system-mode.

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/): 
http://127.0.0.1:180/  gitlab address
Please enter the gitlab-ci token for this runner:
yDNMAooQTxsrLrHgFf6C # token, see `http:/<gitlab host>: 180/admin/runners'.`
Please enter the gitlab-ci description for this runner:
[host1]:
Please enter the gitlab-ci tags for this runner (comma separated):
maven # The label name is `maven'.`
Whether to run untagged builds [true/false]:
[false]:  # Whether to use label driver, that is, to use this `runner'when the label is satisfied`
Whether to lock Runner to current project [true/false]:
[false]:  # Whether or not the current project is exclusive, the above token is global, and another kind of token is associated with the project. If you use the token associated with a project and this is true, it means that the `runner'is used only for the project.
Registering runner... succeeded                     runner=yDNMAooQ
Please enter the executor: kubernetes, docker, parallels, shell, ssh, docker-ssh+machine, docker-ssh, virtualbox, docker+machine:
docker # Actuator type, where `docker'is used`
Please enter the default Docker image (e.g. ruby:2.1):
maven:alpine  # ` docker `container, using maven container here
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

[root[@host1](https://my.oschina.net/host1) target]# gitlab-ci-multi-runner register
Running in system-mode.

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
http://127.0.0.1:180/
Please enter the gitlab-ci token for this runner:
yDNMAooQTxsrLrHgFf6C
Please enter the gitlab-ci description for this runner:
[host1]:
Please enter the gitlab-ci tags for this runner (comma separated):
shell # The label name is `shell'.`
Whether to run untagged builds [true/false]:
[false]:
Whether to lock Runner to current project [true/false]:
[false]:
Registering runner... succeeded                     runner=yDNMAooA
Please enter the executor: kubernetes, docker, parallels, shell, ssh, docker-ssh+machine, docker-ssh, virtualbox, docker+machine:
shell # Actuator type, where `shell'is used`
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

At this point in gitlab, you can see that runner is in effect:

The configuration generated after completion is as follows:

[root[@host1](https://my.oschina.net/host1) target]# cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0

[[runners]]
  name = "host1"
  url = "http://127.0.0.1:180/"
  token = "dfc2643e12d0be2168950490c46999"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "maven:alpine"
    privileged = false
    disable_cache = false
    volumes = ["/cache", "/root/m2:/root/.m2"]  # Modify this to map the maven `m2'directory locally
    pull_policy = "if-not-present" # Add this line, default to pull mirror every execution, change to pull only when nonexistent
    shm_size = 0
  [runners.cache]

[[runners]]
  name = "host1"
  url = "http://127.0.0.1:180/"
  token = "f069825bc10401c71e064218ba3987"
  executor = "shell"
  [runners.cache]

> Tip > > <https://docs.gitlab.com/runner/executors/README.html>;

Install SonarQube.

# Running Sonarqube-dependent databases
docker run --name sonar-db -d \
  -e POSTGRES_USER=sonar \
  -e POSTGRES_PASSWORD=sonar \
  -e POSTGRES_DB=sonar \
  -v /data/sonar-db:/var/lib/postgresql/data \
  postgres:alpine

# Running Sonarquble
docker run --name sonarqube -d \
  -p 9000:9000 -p 9092:9092 \
  -e SONARQUBE_JDBC_USERNAME=sonar \
  -e SONARQUBE_JDBC_PASSWORD=sonar \
  -e SONARQUBE_JDBC_URL=jdbc:postgresql://sonar-db/sonar?characterEncoding=utf8 \
  -v /data/sonarqube/data:/opt/sonarqube/data \
  -v /data/sonarqube/conf:/opt/sonarqube/conf \
  -v /data/sonarqube/extensions:/opt/sonarqube/extensions \  # Mapping Plug-in Directory
  --link sonar-db:sonar-db \
  sonarqube:alpine

Create the plugins directory under / data/sonarqube/extensions and download https://github.com/gabrie-allaigre/sonar-gitlab-plugin/releases/download/2.0.1/sonar-gitlab-plugin-2.0.1.jar to this directory.

Open http:/< host>:9000/updatecenter/available username/password: `admin/admin', download SonarJava or other plug-ins and restart

Configure several parameters according to https://gitlab.talanlabs.com/gabriel-allaigre/sonar-gitlab-plugin#configuration

Example Engineering

/src/main/java/com/ecfront/test/devops/DevOpsApplication.java.

package com.ecfront.test.devops;

import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;

@SpringBootApplication
public class DevOpsApplication {

    public static void main(String[] args) {
        new SpringApplicationBuilder(DevOpsApplication.class).web(true).run(args); 
    }

}

The Spring Boot example starts a web service with port 8080 by default

/src/main/resources/application.yml.

spring:
  application:
    name: DevOps

Specify component name, negligible

/.gitlab-ci.yml.

stages:  # Define three stages (pipeline)
  - verify
  - build
  - deploy

maven-build:  # job name, optional name
  image: maven:alpine # Running with a maven container is negligible
  stage: build  # Binding this job to the `build'phase
  only: 
   - master # Execution is triggered only when master branch changes
  tags: 
   - maven # Use `runner'with the label `maven'.
  script: "mvn package -B"  # Execute the script, here is the packaging operation
  artifacts: 
    paths:
      - target/*.jar # This stage outputs the file, where the packaged fatjar is exported

docker-deploy:
  stage: deploy 
  tags: 
   - shell # This phase is executed using `runner'labeled `shell'.
  script:  # The script logic here is to first compile the file containing the jar into a mirror, and then run it
  - sudo docker build -t ecfront/test:1.0 .
  - app="test"
  - if sudo docker ps | awk -v app="app" 'NR>1{  ($(NF) == app )  }'; then
  -  sudo docker stop "$app" && sudo docker rm -f "$app"
  - fi
  - sudo docker run --name test -d -p 8080:8080 ecfront/test:1.0

sonarqube:  # Configuration of sonarqube
  stage: verify
  only:
    - master
  tags:
    - maven
  script: # Running sonarqube processing with mvn, $SONAR\url is the url of sonarqube
    - mvn verify sonar:sonar -Dsonar.host.url=$SONAR_URL

> Tip > > Add SONAR_URL=< sonarqube url&gt to the variable. > > See: https://docs.gitlab.com/ce/ci/variables/

> Tip > > <https://docs.gitlab.com/ce/ci/yaml/README.html>;

/Dockerfile

FROM java:8-jdk-alpine 
VOLUME /tmp
ADD /target/devops.jar devops.jar 
ENTRYPOINT ["java","-jar","/devops.jar"]

Running with a java container
Add the bags typed above to the mirror
After the container runs, it runs the corresponding jar file

/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.3.RELEASE</version>
    </parent>

    <groupId>com.ecfront.test</groupId>
    <artifactId>devops</artifactId>
    <packaging>jar</packaging>
    <version>1.0.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.sonarsource.scanner.maven</groupId>
            <artifactId>sonar-maven-plugin</artifactId>
            <version>3.3.0.603</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <finalName>${project.artifactId}</finalName>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>