WeChat Public Number: Operations and Maintenance Development Story, Author: double winter
1. Familiarize yourself with the Spring Cloud microservice project
Code branch description:
-
dev1 Delivery Code
-
dev2 Writes Dockerfile to build a mirror
-
dev3 K8S Resource Layout
-
dev4 Micro-Service Link Monitoring
-
master finally goes online
2. Logical architecture for deploying Spring Cloud micro service project in K8S
Overall Logical Architecture Diagram
Diagram of service exposure
3. Prepare the environment
A set of k8s clusters, either single master or multiple masters
| IP | role | To configure |
|---|---|---|
| 192.168.73.138 | master | 2C4G |
| 192.168.73.139 | node1 | 2C4G |
| 192.168.73.140 | node2 | 2C4G |
| 192.168.73.137 | harbor,mysql | 1C2G |
-
CoreDNS
-
Ingress Controller (refer to previous k8s build files)
-
Preparing mariadb database
192.168 outside k8s. Install mariadb on 73.137
[root@localhost ~]# yum -y install mariadb mariadb-server [root@localhost ~]# systemctl start mariadb && systemctl enable mariadb Created symlink from /etc/systemd/system/multi-user.target.wants/mariadb.service to /usr/lib/systemd/system/mariadb.service. [root@localhost ~]# mysql_secure_installation ###Initialize the database, except to set the password to 123 456, all else can be returned Reference article: https://www.jianshu.com/p/85ad52c88399
4. Source code building
The Maven Project Object Model (POM) allows you to manage project management tool software for project building, reporting, and documentation using a short description
Install jdk and maven environments
[root@k8s-master ~]# yum install java-1.8.0-openjdk maven -y [root@k8s-master ~]# java -version openjdk version "1.8.0_242" OpenJDK Runtime Environment (build 1.8.0_242-b08) OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode) [root@k8s-master ~]# mvn -version Apache Maven 3.0.5 (Red Hat 3.0.5-17) Maven home: /usr/share/maven Java version: 1.8.0_242, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-957.el7.x86_64", arch: "amd64", family: "unix"
Setting up maven's endoscopic image source
/etc/maven/settings.xml
<mirror> <id>nexus-aliyun</id> <mirrorOf>*</mirrorOf> <name>Nexus aliyun</name> <url>http://maven.aliyun.com/nexus/content/groups/public</url> </mirror>
Pull Substitution Code
[root@k8s-master ~]# cd /opt [root@k8s-master opt]# git clone https://github.com/lizhenliang/simple-microservice.git [root@k8s-master opt]# cd simple-microservice/ [root@k8s-master simple-microservice]# ls basic-common eureka-service k8s lombok.config pom.xml product-service stock-service db gateway-service LICENSE order-service portal-service README.md
Code Compilation Build Test
[root@k8s-master simple-microservice]# mvn clean package -Dmaven.test.skip=true .............. [INFO] Reactor Summary: [INFO] [INFO] simple-microservice ............................... SUCCESS [0.164s] [INFO] basic-common ...................................... SUCCESS [0.008s] [INFO] basic-common-core ................................. SUCCESS [4.749s] [INFO] gateway-service ................................... SUCCESS [5:54.575s] [INFO] eureka-service .................................... SUCCESS [29.727s] [INFO] product-service ................................... SUCCESS [0.005s] [INFO] product-service-api ............................... SUCCESS [0.987s] [INFO] stock-service ..................................... SUCCESS [0.003s] [INFO] stock-service-api ................................. SUCCESS [0.808s] [INFO] product-service-biz ............................... SUCCESS [11.105s] [INFO] stock-service-biz ................................. SUCCESS [1.290s] [INFO] order-service ..................................... SUCCESS [0.002s] [INFO] order-service-api ................................. SUCCESS [1.319s] [INFO] order-service-biz ................................. SUCCESS [2.531s] [INFO] basic-common-bom .................................. SUCCESS [0.002s] [INFO] portal-service .................................... SUCCESS [3.707s] ..............
5. Build Project Mirror Push to Mirror Warehouse
There is a Dockerfile under each service of the project, which can be used to build the project image. We can go to the gateway-service directory of the gateway service and check the file. This demo project is already written, you can build the image directly, and then pack it and push it to the mirror warehouse
[root@k8s-master simple-microservice]# cd gateway-service/
[root@k8s-master gateway-service]# ls
Dockerfile pinpoint pom.xml src target
[root@k8s-master gateway-service]# cat Dockerfile
FROM lizhenliang/java:8-jdk-alpine
LABEL maintainer www.ctnrs.com
ENV JAVA_ARGS="-Dfile.encoding=UTF8 -Duser.timezone=GMT+08"
COPY ./target/gateway-service.jar ./
COPY pinpoint /pinpoint
EXPOSE 9999
CMD java -jar -javaagent:/pinpoint/pinpoint-bootstrap-1.8.3.jar -Dpinpoint.agentId=$(echo $HOSTNAME | awk -F- '{print "gateway-"$NF}') -Dpinpoint.applicationName=ms-gateway $JAVA_ARGS $JAVA_OPTS /gateway-service.jar
Go back to the previous level and into the k8s directory, you will see a script that builds a mirror and pushes it to the mirror warehouse, wrapping each item in a loop, and then pushing it to the harbor mirror warehouse, where the warehouse address needs to be modified to its own address, and then go to the harbor mirror warehouse to create a microservice project
[root@k8s-master gateway-service]# cd ..
[root@k8s-master simple-microservice]# cd k8s/
[root@k8s-master k8s]# ls
docker_build.sh eureka.yaml gateway.yaml order.yaml portal.yaml product.yaml stock.yaml
[root@k8s-master k8s]# cat docker_build.sh
#!/bin/bash
kubectl create ns ms
docker_registry=192.168.73.137
kubectl create ns ms
kubectl create secret docker-registry registry-pull-secret --docker-server=$docker_registry --docker-username=admin --docker-password=Harbor12345 --docker-email=admin@ctnrs.com -n ms
service_list="eureka-service gateway-service order-service product-service stock-service portal-service"
service_list=${1:-${service_list}}
work_dir=$(dirname $PWD)
current_dir=$PWD
cd $work_dir
mvn clean package -Dmaven.test.skip=true
for service in $service_list; do
cd $work_dir/$service
if ls |grep biz &>/dev/null; then
cd ${service}-biz
fi
service=${service%-*}
image_name=$docker_registry/microservice/${service}:$(date +%F-%H-%M-%S)
docker build -t ${image_name} .
docker push ${image_name}
done
[root@k8s-master k8s]# ./docker_build.sh
6.K8S Service Layout
- Gateway service gateway, pod creation using Deployment, expose services using ingress
[root@k8s-master k8s]# cat gateway.yaml
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gateway
namespace: ms
spec:
rules:
- host: gateway.ctnrs.com
http:
paths:
- path: /
backend:
serviceName: gateway
servicePort: 9999
---
apiVersion: v1
kind: Service
metadata:
name: gateway
namespace: ms
spec:
ports:
- port: 9999
name: gateway
selector:
project: ms
app: gateway
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: gateway
namespace: ms
spec:
replicas: 2
selector:
matchLabels:
project: ms
app: gateway
template:
metadata:
labels:
project: ms
app: gateway
spec:
imagePullSecrets:
- name: registry-pull-secret
containers:
- name: gateway
image: 192.168.73.137/microservice/gateway:2020-03-08-16-40-54
imagePullPolicy: Always
ports:
- protocol: TCP
containerPort: 9999
env:
- name: JAVA_OPTS
value: "-Xmx1g"
resources:
requests:
cpu: 0.5
memory: 256Mi
limits:
cpu: 1
memory: 1Gi
readinessProbe:
tcpSocket:
port: 9999
initialDelaySeconds: 60
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 9999
initialDelaySeconds: 60
periodSeconds: 10
- Front-end portal
[root@k8s-master k8s]# cat product.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: portal
namespace: ms
spec:
rules:
- host: portal.ctnrs.com
http:
paths:
- path: /
backend:
serviceName: portal
servicePort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: portal
namespace: ms
spec:
ports:
- port: 8080
name: portal
selector:
project: ms
app: portal
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: portal
namespace: ms
spec:
replicas: 1
selector:
matchLabels:
project: ms
app: portal
template:
metadata:
labels:
project: ms
app: portal
spec:
imagePullSecrets:
- name: registry-pull-secret
containers:
- name: portal
image: 192.168.73.137/microservice/portal:2020-03-08-16-41-31
imagePullPolicy: Always
ports:
- protocol: TCP
containerPort: 8080
env:
- name: JAVA_OPTS
value: "-Xmx1g"
resources:
requests:
cpu: 0.5
memory: 256Mi
limits:
cpu: 1
memory: 1Gi
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 60
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 60
periodSeconds: 10
- yaml of inventory service stock
[root@k8s-master k8s]# cat stock.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: stock
namespace: ms
spec:
replicas: 2
selector:
matchLabels:
project: ms
app: stock
template:
metadata:
labels:
project: ms
app: stock
spec:
imagePullSecrets:
- name: registry-pull-secret
containers:
- name: stock
image: 192.168.73.137/microservice/stock:2020-03-08-16-41-23
imagePullPolicy: Always
ports:
- protocol: TCP
containerPort: 8030
env:
- name: JAVA_OPTS
value: "-Xmx1g"
resources:
requests:
cpu: 0.5
memory: 256Mi
limits:
cpu: 1
memory: 1Gi
readinessProbe:
tcpSocket:
port: 8030
initialDelaySeconds: 60
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 8030
initialDelaySeconds: 60
periodSeconds: 10
- yaml file for order service
[root@k8s-master k8s]# cat order.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: order
namespace: ms
spec:
replicas: 2
selector:
matchLabels:
project: ms
app: order
template:
metadata:
labels:
project: ms
app: order
spec:
imagePullSecrets:
- name: registry-pull-secret
containers:
- name: order
image: 192.168.73.137/microservice/order:2020-03-08-16-41-04
imagePullPolicy: Always
ports:
- protocol: TCP
containerPort: 8020
env:
- name: JAVA_OPTS
value: "-Xmx1g"
resources:
requests:
cpu: 0.5
memory: 256Mi
limits:
cpu: 1
memory: 1Gi
readinessProbe:
tcpSocket:
port: 8020
initialDelaySeconds: 60
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 8020
initialDelaySeconds: 60
periodSeconds: 10
- yaml file for commodity service product
[root@k8s-master k8s]# cat product.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: product
namespace: ms
spec:
replicas: 2
selector:
matchLabels:
project: ms
app: product
template:
metadata:
labels:
project: ms
app: product
spec:
imagePullSecrets:
- name: registry-pull-secret
containers:
- name: product
image: 192.168.73.137/microservice/product:2020-03-08-16-41-14
imagePullPolicy: Always
ports:
- protocol: TCP
containerPort: 8010
env:
- name: JAVA_OPTS
value: "-Xmx1g"
resources:
requests:
cpu: 0.5
memory: 256Mi
limits:
cpu: 1
memory: 1Gi
readinessProbe:
tcpSocket:
port: 8010
initialDelaySeconds: 60
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 8010
initialDelaySeconds: 60
periodSeconds: 10
- yaml file for registry eureka
Deployed with a stateful controller StatefulSet and exposed service headless services for access using an ingress controller
[root@k8s-master k8s]# cat eureka.yaml
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: eureka
namespace: ms
spec:
rules:
- host: eureka.ctnrs.com
http:
paths:
- path: /
backend:
serviceName: eureka
servicePort: 8888
---
apiVersion: v1
kind: Service
metadata:
name: eureka
namespace: ms
spec:
clusterIP: None
ports:
- port: 8888
name: eureka
selector:
project: ms
app: eureka
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: eureka
namespace: ms
spec:
replicas: 3
selector:
matchLabels:
project: ms
app: eureka
serviceName: "eureka"
template:
metadata:
labels:
project: ms
app: eureka
spec:
imagePullSecrets:
- name: registry-pull-secret
containers:
- name: eureka
image: 192.168.73.137/microservice/eureka:2020-03-08-16-40-31
ports:
- protocol: TCP
containerPort: 8888
env:
- name: JAVA_OPTS
value: "-Xmx1g"
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
resources:
requests:
cpu: 0.5
memory: 256Mi
limits:
cpu: 1
memory: 1Gi
readinessProbe:
tcpSocket:
port: 8888
initialDelaySeconds: 60
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 8888
initialDelaySeconds: 60
periodSeconds: 10
7. Deploy Eureka Cluster (Registry) in K8S
Manual deployment using prepared yaml files
❝
Note the modifications, mirror address in yaml
❞
[root@k8s-master simple-microservice]# cd k8s/ [root@k8s-master k8s]# kubectl apply -f eureka.yaml ingress.extensions/eureka created service/eureka created statefulset.apps/eureka created [root@k8s-master k8s]# kubectl get pod,svc,ingress -n ms -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/eureka-0 1/1 Running 1 3h51m 172.17.10.4 192.168.73.140 <none> <none> pod/eureka-1 1/1 Running 0 3h47m 172.17.10.5 192.168.73.140 <none> <none> pod/eureka-2 1/1 Running 0 149m 172.17.42.3 192.168.73.135 <none> <none> NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/eureka ClusterIP None <none> 8888/TCP 3h51m app=eureka,project=ms NAME HOSTS ADDRESS PORTS AGE ingress.extensions/eureka eureka.ctnrs.com 80 3h51m
For Eureka in the local host file. Ctnrs. COM domain name, then browse to access
Registration Center
8. Import database files into MySQL database
On a database server:
[root@localhost ~]# scp -r root@192.168.73.138:/opt/simple-microservice/db . [root@localhost ~]# mysql -uroot -p Enter password: mysql>use mysql; MariaDB [mysql]> update user set host = '%' where user = 'root' and host='localhost'; Query OK, 1 row affected (0.07 sec) Rows matched: 1 Changed: 1 Warnings: 0 MariaDB [mysql]> select host, user from user; +-----------+------+ | host | user | +-----------+------+ | % | root | | 127.0.0.1 | root | | ::1 | root | +-----------+------+ 3 rows in set (0.00 sec) MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | +--------------------+ 3 rows in set (0.01 sec) MariaDB [(none)]> create database tb_order; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> create database tb_product; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> create database tb_stock; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | tb_order | | tb_product | | tb_stock | +--------------------+ 6 rows in set (0.01 sec) MariaDB [(none)]> use tb_order; Database changed MariaDB [tb_order]> source ~/db/order.sql; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 1 row affected (0.01 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) MariaDB [tb_order]> use tb_product; Database changed MariaDB [tb_product]> source ~/db/product.sql; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.01 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 1 row affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 4 rows affected (0.00 sec) Records: 4 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) MariaDB [tb_product]> use tb_stock; Database changed MariaDB [tb_stock]> source ~/db/stock.sql; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 1 row affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 4 rows affected (0.00 sec) Records: 4 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.01 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) MariaDB [tb_stock]>
9. Deployment gateway
❝
Note that modifying the mirror address in yaml
❞
[root@k8s-master k8s]# kubectl create -f gateway.yaml ingress.extensions/gateway created service/gateway created deployment.apps/gateway created [root@k8s-master k8s]# kubectl get pod -n ms -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES eureka-0 1/1 Running 1 4h28m 172.17.10.4 192.168.73.140 <none> <none> eureka-1 1/1 Running 2 4h24m 172.17.10.5 192.168.73.140 <none> <none> eureka-2 1/1 Running 0 3h6m 172.17.42.3 192.168.73.135 <none> <none> gateway-7877fc8867-gsb59 1/1 Running 0 9m32s 172.17.42.2 192.168.73.135 <none> <none> gateway-7877fc8867-rbxqz 1/1 Running 0 9m32s 172.17.5.2 192.168.73.138 <none> <none>
10. Deploy business programs (product, stock, order)
Attention needs to be paid to modifying the database addresses in the three business program profiles and the mirror addresses in yaml
[root@k8s-master k8s]# vim ../product-service/product-service-biz/src/main/resources/application-fat.yml
spring:
datasource:
url: jdbc:mysql://192.168.73.137:3306/tb_product?characterEncoding=utf-8
username: root
password: 123456
driver-class-name: com.mysql.jdbc.Driver
eureka:
instance:
prefer-ip-address: true
client:
register-with-eureka: true
fetch-registry: true
service-url:
defaultZone: http://eureka-0.eureka.ms:8888/eureka,http://eureka-1.eureka.ms:8888/eureka,http://eureka-2.eureka.ms:8888/eureka
[root@k8s-master k8s]# kubectl create -f product.yaml
deployment.apps/product created
[root@k8s-master k8s]# vim ../stock-service/stock-service-biz/src/main/resources/application-fat.yml
spring:
datasource:
url: jdbc:mysql://192.168.73.137:3306/tb_stock?characterEncoding=utf-8
username: root
password: 123456
driver-class-name: com.mysql.jdbc.Driver
eureka:
instance:
prefer-ip-address: true
client:
register-with-eureka: true
fetch-registry: true
service-url:
defaultZone: http://eureka-0.eureka.ms:8888/eureka,http://eureka-1.eureka.ms:8888/eureka,http://eureka-2.eureka.ms:8888/eureka
[root@k8s-master k8s]# kubectl create -f stock.yaml
deployment.apps/stock created
[root@k8s-master k8s]# vim ../order-service/order-service-biz/src/main/resources/application-fat.yml
spring:
datasource:
url: jdbc:mysql://192.168.73.137:3306/tb_order?characterEncoding=utf-8
username: root
password: 123456
driver-class-name: com.mysql.jdbc.Driver
eureka:
instance:
prefer-ip-address: true
client:
register-with-eureka: true
fetch-registry: true
service-url:
defaultZone: http://eureka-0.eureka.ms:8888/eureka,http://eureka-1.eureka.ms:8888/eureka,http://eureka-2.eureka.ms:8888/eureka
[root@k8s-master k8s]# kubectl create -f order.yaml
deployment.apps/order created
- Check pod status
It is only normal that all pods are in the running state. When other states occur, you need to use the r command to sort out, properly reduce pods, and increase the health check-up time
-
kubectl get pod -o yaml #Check if Pod is configured correctly
-
kubectl describe pod #View Pod details
-
kubectl logs [-c] #View container logs
[root@k8s-master k8s]# kubectl get pod -n ms -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES eureka-0 1/1 Running 1 5h11m 172.17.10.4 192.168.73.140 <none> <none> eureka-1 1/1 Running 2 7s 172.17.10.5 192.168.73.140 <none> eureka-2 0/1 Running 1 3m45s 172.17.42.2 192.168.73.135 <none> <none> gateway-7d9c68f9b9-597b8 1/1 Running 0 7m28s 172.17.42.3 192.168.73.135 <none> <none> product-5bb494596b-5m75d 1/1 Running 0 16s 172.17.10.3 192.168.73.140 <none> <none> product-845889bccb-9z982 1/1 Running 10 36m 172.17.42.4 192.168.73.135 <none> <none> stock-7fdb87cf6c-wsp4r 1/1 Running 9 32m 172.17.42.5 192.168.73.135 <none> <none>
11. Deployment Front End (portal)
❝
Note that modifying the mirror address in yaml
❞
[root@k8s-master k8s]# kubectl create -f portal.yaml ingress.extensions/portal created service/portal created deployment.apps/portal created [root@k8s-master k8s]# kubectl get pod -n ms -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES eureka-0 1/1 Running 1 5h11m 172.17.10.4 192.168.73.140 <none> <none> eureka-1 1/1 Running 2 7s 172.17.10.5 192.168.73.140 <none> eureka-2 0/1 Running 1 3m45s 172.17.42.2 192.168.73.135 <none> <none> gateway-7d9c68f9b9-597b8 1/1 Running 0 7m28s 172.17.42.3 192.168.73.135 <none> <none> product-5bb494596b-5m75d 1/1 Running 0 16s 172.17.10.3 192.168.73.140 <none> <none> product-845889bccb-9z982 1/1 Running 10 36m 172.17.42.4 192.168.73.135 <none> <none> stock-7fdb87cf6c-wsp4r 1/1 Running 9 32m 172.17.42.5 192.168.73.135 <none> <none> portal-cc6844699-4wr9j 1/1 Running 1 46m 172.17.10.2 192.168.73.140 <none> <none>
12. Access Front End Authentication
For portal in the local host file. Ctnrs. COM domain name, then browse to access
Registration Center
Front end
Commodity Services
Commodity Services
Order service is warm
If you like this article, please share it with your circle of friends and follow me for more information.