A Demo of using Metallb to implement LoadBalancer in Kubernetes Service

Write in front

  • Learn the problems encountered in K8s, sort them out and share them with your friends
  • The content of the blog involves:
  • Metallb creation
  • Creation of a Service of type LoadBalancer

Some people's hearts are like flowers and trees, all born to the sun -- the war fire drama Princes "sword coming"

Service load balancing problem

NodePort has not completely solved all the problems of external access to services, such as load balancing. If there are 10 nodes in our cluster, it is best to have a load balancer at this time. External requests only need to access the IP address of the load balancer, and the load balancer is responsible for forwarding traffic to the NodePort of a later Node. As shown in the figure

Load balancing of NodePort

LoadBalancer mode

The Load balancer component is independent of the Kubernetes cluster. It is usually a hardware Load balancer or implemented in software, such as HAProxy or Nginx. For each Service, we usually need to configure a corresponding Load balancer instance to forward traffic to the back-end Node

Kubernetes provides an automated solution. If our cluster runs on Google's GCE public cloud, as long as we change the type nodeport of the Service to type loadbalancer, kubernetes will automatically create a corresponding Load balancer instance and return its IP address for external clients. When there is no public cloud, we can also use some plug-ins, such as metallb

The LoadBalancer needs to establish a load pool outside the Service. Then assign an IP to the Service.

When we directly create a LoadBalancer Service, it will always be in the pending state because we do not have a corresponding cloud load balancer

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$kubectl expose  --name=blogsvc pod blog --port=80 --type=LoadBalancer
service/blogsvc exposed
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$kubectl  get svc -o wide | grep blogsvc
blogsvc   LoadBalancer   10.106.28.175   <pending>     80:32745/TCP   26s   run=blog
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$

Metallb create

Metallb can provide LB type Service support through k8s native mode

Use: metallb https://metallb.universe.tf/
resource file https://github.com/metallb/metallb/blob/main/manifests/metallb.yaml

Create namespace

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$kubectl  create  ns metallb-system
namespace/metallb-system created
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$kubectl  config set-context  $(kubectl config current-context) --namespace=metallb-system
Context "kubernetes-admin@kubernetes" modified.
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create]
└─$

: set paste solves the problem of paste confusion

Create metallb

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  apply  -f metallb.yaml
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  get pods -o wide
NAME                         READY   STATUS    RESTARTS   AGE     IP               NODE                         NOMINATED NODE   READINESS GATES
controller-66d9554cc-8rxq8   1/1     Running   0          3m36s   10.244.171.170   vms82.liruilongs.github.io   <none>           <none>
speaker-bbl94                1/1     Running   0          3m36s   192.168.26.83    vms83.liruilongs.github.io   <none>           <none>
speaker-ckbzj                1/1     Running   0          3m36s   192.168.26.81    vms81.liruilongs.github.io   <none>           <none>
speaker-djmpr                1/1     Running   0          3m36s   192.168.26.82    vms82.liruilongs.github.io   <none>           <none>
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$

Create address pool - 192.168 26.240-192.168. twenty-six point two five zero

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$vim pool.yaml
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl apply  -f pool.yaml
configmap/config created

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.26.240-192.168.26.250

192.168 is allocated through metallb using the configuration of type=LoadBalancer 26.240 this address is for blogsvc

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  get svc
No resources found in metallb-system namespace.
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl config  set-context  $(kubectl config current-context) --namespace=liruilong-svc-create
Context "kubernetes-admin@kubernetes" modified.

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  get svc
NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
dbsvc   ClusterIP   10.102.137.59   <none>        3306/TCP   101m
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  expose  --name=blogsvc pod blog --port=80 --type=LoadBalancer
service/blogsvc exposed
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  get svc -o wide
NAME      TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)        AGE    SELECTOR
blogsvc   LoadBalancer   10.108.117.197   192.168.26.240   80:30230/TCP   9s     run=blog
dbsvc     ClusterIP      10.102.137.59    <none>           3306/TCP       101m   run=dbpod
Direct access to 192.168 26.240 will do

It can also be accessed after creating one

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  expose  --name=blogsvc-1 pod blog --port=80 --type=LoadBalancer
service/blogsvc-1 exposed
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$kubectl  get svc -o wide
NAME        TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)        AGE    SELECTOR
blogsvc     LoadBalancer   10.108.117.197   192.168.26.240   80:30230/TCP   11m    run=blog
blogsvc-1   LoadBalancer   10.110.58.143    192.168.26.241   80:31827/TCP   3s     run=blog
dbsvc       ClusterIP      10.102.137.59    <none>           3306/TCP       113m   run=dbpod
┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$
You can also access

Related resource files

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$cat namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: metallb-system
  labels:
    app: metallb

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$cat pool.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.26.240-192.168.26.250


┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-svc-create/metalld]
└─$cat metallb.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  labels:
    app: metallb
  name: controller
  namespace: metallb-system
spec:
  allowPrivilegeEscalation: false
  allowedCapabilities: []
  allowedHostPaths: []
  defaultAddCapabilities: []
  defaultAllowPrivilegeEscalation: false
  fsGroup:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  hostIPC: false
  hostNetwork: false
  hostPID: false
  privileged: false
  readOnlyRootFilesystem: true
  requiredDropCapabilities:
  - ALL
  runAsUser:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    ranges:
    - max: 65535
      min: 1
    rule: MustRunAs
  volumes:
  - configMap
  - secret
  - emptyDir
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  labels:
    app: metallb
  name: speaker
  namespace: metallb-system
spec:
  allowPrivilegeEscalation: false
  allowedCapabilities:
  - NET_RAW
  allowedHostPaths: []
  defaultAddCapabilities: []
  defaultAllowPrivilegeEscalation: false
  fsGroup:
    rule: RunAsAny
  hostIPC: false
  hostNetwork: true
  hostPID: false
  hostPorts:
  - max: 7472
    min: 7472
  - max: 7946
    min: 7946
  privileged: true
  readOnlyRootFilesystem: true
  requiredDropCapabilities:
  - ALL
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - configMap
  - secret
  - emptyDir
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: metallb
  name: controller
  namespace: metallb-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: metallb
  name: speaker
  namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: metallb
  name: metallb-system:controller
rules:
- apiGroups:
  - ''
  resources:
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ''
  resources:
  - services/status
  verbs:
  - update
- apiGroups:
  - ''
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - policy
  resourceNames:
  - controller
  resources:
  - podsecuritypolicies
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: metallb
  name: metallb-system:speaker
rules:
- apiGroups:
  - ''
  resources:
  - services
  - endpoints
  - nodes
  verbs:
  - get
  - list
  - watch
- apiGroups: ["discovery.k8s.io"]
  resources:
  - endpointslices
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ''
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - policy
  resourceNames:
  - speaker
  resources:
  - podsecuritypolicies
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app: metallb
  name: config-watcher
  namespace: metallb-system
rules:
- apiGroups:
  - ''
  resources:
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app: metallb
  name: pod-lister
  namespace: metallb-system
rules:
- apiGroups:
  - ''
  resources:
  - pods
  verbs:
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app: metallb
  name: controller
  namespace: metallb-system
rules:
- apiGroups:
  - ''
  resources:
  - secrets
  verbs:
  - create
- apiGroups:
  - ''
  resources:
  - secrets
  resourceNames:
  - memberlist
  verbs:
  - list
- apiGroups:
  - apps
  resources:
  - deployments
  resourceNames:
  - controller
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: metallb
  name: metallb-system:controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metallb-system:controller
subjects:
- kind: ServiceAccount
  name: controller
  namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: metallb
  name: metallb-system:speaker
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metallb-system:speaker
subjects:
- kind: ServiceAccount
  name: speaker
  namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app: metallb
  name: config-watcher
  namespace: metallb-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: config-watcher
subjects:
- kind: ServiceAccount
  name: controller
- kind: ServiceAccount
  name: speaker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app: metallb
  name: pod-lister
  namespace: metallb-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-lister
subjects:
- kind: ServiceAccount
  name: speaker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app: metallb
  name: controller
  namespace: metallb-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: controller
subjects:
- kind: ServiceAccount
  name: controller
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: metallb
    app.kubernetes.io/component: speaker
  name: speaker
  namespace: metallb-system
spec:
  selector:
    matchLabels:
      app: metallb
      app.kubernetes.io/component: speaker
  template:
    metadata:
      annotations:
        prometheus.io/port: '7472'
        prometheus.io/scrape: 'true'
      labels:
        app: metallb
        app.kubernetes.io/component: speaker
    spec:
      containers:
      - args:
        - --port=7472
        - --config=config
        - --log-level=info
        env:
        - name: METALLB_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: METALLB_HOST
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: METALLB_ML_BIND_ADDR
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        # needed when another software is also using memberlist / port 7946
        # when changing this default you also need to update the container ports definition
        # and the PodSecurityPolicy hostPorts definition
        #- name: METALLB_ML_BIND_PORT
        #  value: "7946"
        - name: METALLB_ML_LABELS
          value: "app=metallb,app.kubernetes.io/component=speaker"
        - name: METALLB_ML_SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: memberlist
              key: secretkey
        image: quay.io/metallb/speaker:main
        name: speaker
        ports:
        - containerPort: 7472
          name: monitoring
        - containerPort: 7946
          name: memberlist-tcp
        - containerPort: 7946
          name: memberlist-udp
          protocol: UDP
        livenessProbe:
          httpGet:
            path: /metrics
            port: monitoring
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /metrics
            port: monitoring
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_RAW
            drop:
            - ALL
          readOnlyRootFilesystem: true
      hostNetwork: true
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: speaker
      terminationGracePeriodSeconds: 2
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: metallb
    app.kubernetes.io/component: controller
  name: controller
  namespace: metallb-system
spec:
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: metallb
      app.kubernetes.io/component: controller
  template:
    metadata:
      annotations:
        prometheus.io/port: '7472'
        prometheus.io/scrape: 'true'
      labels:
        app: metallb
        app.kubernetes.io/component: controller
    spec:
      containers:
      - args:
        - --port=7472
        - --config=config
        - --log-level=info
        env:
        - name: METALLB_ML_SECRET_NAME
          value: memberlist
        - name: METALLB_DEPLOYMENT
          value: controller
        image: quay.io/metallb/controller:main
        name: controller
        ports:
        - containerPort: 7472
          name: monitoring
        livenessProbe:
          httpGet:
            path: /metrics
            port: monitoring
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /metrics
            port: monitoring
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - all
          readOnlyRootFilesystem: true
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        fsGroup: 65534
      serviceAccountName: controller
      terminationGracePeriodSeconds: 0

Keywords: Kubernetes Load Balance Container

Added by leetee on Sat, 01 Jan 2022 16:28:29 +0200

Popular Keywords

©2024 Programming VIP