1. Data indicators of Prometheus are obtained through an open HTTP(S) data interface. We do not need to install monitoring agent s separately. We only need to expose a metrics interface. Prometheus will pull data regularly. For some ordinary HTTP services, we can reuse this service directly. Add A / metrics interface to expose Prometheus
2. Some services, even without native integration of the interface, can fully use exporters to obtain index data, such as mysqld_exporter, node_exporter, redis-exporter. These exporters are somewhat similar to agent s in traditional monitoring services, which collect target service index data and then directly explode. To Prometheus.

Monitoring Traefik with metric interface
1. Modify its configuration file traefik.toml, add the following, and open the metirc interface

[metrics]
  [metrics.prometheus]
    entryPoint = "traefik"
    buckets = [0.1, 0.3, 1.2, 5.0]

2. Then update traefik configmap and traefik pod
$ kubectl get configmap -n kube-system
traefik-conf 1 83d
$ kubectl delete configmap traefik-conf -n kube-system
$ kubectl create configmap traefik-conf --from-file=traefik.toml -n kube-system
$ kubectl apply -f traefik.yaml
$ kubectl get svc -n kube-system |grep traefik
traefik-ingress-service NodePort 10.100.222.78 <none> 80:31657/TCP,8080:31572/TCP
$ curl 10.100.222.78:8080/metrics
$ curl 192.168.1.243:31572/metrics
3. Update prometheus configuration file and add job

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: kube-ops
data:
  prometheus.yml: |
    global:
      scrape_interval: 30s
      scrape_timeout: 30s

    scrape_configs:
    - job_name: 'prometheus'
      static_configs:
        - targets: ['localhost:9090']

    - job_name: 'traefik'
      static_configs:
        - targets: ['traefik-ingress-service.kube-system.svc.cluster.local:8080']

Kubectl apply-f prome-cm.yaml # Update prometheus configemap file
Since the service cename corresponding to Traefik here is traefik-ingress-service, and under the namespace of kube-system, the path configuration of targets here needs to use the form of FQDN: traefik-ingress-service.kube-system.svc.cluster.local.
$ kubectl get svc -n kube-ops |grep prometheus
prometheus NodePort 10.102.197.83 <none> 9090:32619/TCP
$ curl -X POST "http://192.168.1.243:32619/-/reload "# Making the configuration effective may take some time, and using reload command can make the configuration effective without updating prometheus pod

Use redis-exporter to monitor redis services
redis-exporter is deployed in the same pod as sidecar and redis
1. Constructing pod and svc
$ docker pull redis:4
$ docker pull oliver006/redis_exporter:latest

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: redis
  namespace: kube-ops
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9121"
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:4
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 6379
      - name: redis-exporter
        image: oliver006/redis_exporter:latest
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 9121
---
kind: Service
apiVersion: v1
metadata:
  name: redis
  namespace: kube-ops
spec:
  selector:
    app: redis
  ports:
  - name: redis
    port: 6379
    targetPort: 6379
  - name: prom
    port: 9121
    targetPort: 9121

$ kubectl get svc -n kube-ops |grep redis
redis ClusterIP 10.105.241.59 <none> 6379/TCP,9121/TCP
curl 10.105.241.59:9121/metrics# to see if it can be monitored
2. Add job and update prometheus configmap configuration file

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: kube-ops
data:
  prometheus.yml: |
    global:
      scrape_interval: 30s
      scrape_timeout: 30s
    scrape_configs:
    - job_name: 'prometheus'
      static_configs:
      - targets: ['localhost:9090']
    - job_name: 'traefik'
      static_configs:
      - targets: ['traefik-ingress-service.kube-system.svc.cluster.local:8080']
    - job_name: 'redis'
      static_configs:
      - targets: ['redis:9121']

Because our redis service and Prometheus are in the same namespace, we use servicename directly.
Kubectl apply-f prometheus-cm.yaml # update configuration
$ kubectl get svc -n kube-ops |grep prometheus
prometheus NodePort 10.102.197.83 <none> 9090:32619/TCP
$ curl -X POST "http://10.102.197.83:9090/-/reload "# Make the configuration effective
http://http://192.168.1.243:32619/targets

Using node-exporter to monitor kubelet of all nodes and each node in k8s cluster
1. Deploying node-exporter
Deploy the service through the DaemonSet controller so that each node automatically runs a Pod like this

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: kube-ops
  labels:
    name: node-exporter
spec:
  template:
    metadata:
      labels:
        name: node-exporter
    spec:
      hostPID: true
      hostIPC: true
      hostNetwork: true
      containers:
      - name: node-exporter
        image: prom/node-exporter:v0.16.0
        ports:
        - containerPort: 9100
        resources:
          requests:
            cpu: 0.15
        securityContext:
          privileged: true
        args:
        - --path.procfs
        - /host/proc
        - --path.sysfs
        - /host/sys
        - --collector.filesystem.ignored-mount-points
        - '"^/(sys|proc|dev|host|etc)($|/)"'
        volumeMounts:
        - name: dev
          mountPath: /host/dev
        - name: proc
          mountPath: /host/proc
        - name: sys
          mountPath: /host/sys
        - name: rootfs
          mountPath: /rootfs
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: dev
          hostPath:
            path: /dev
        - name: sys
          hostPath:
            path: /sys
        - name: rootfs
          hostPath:
            path: /

Since the data we want to get is the monitoring index data of the host, and our node-exporter is running in the container, we need to configure some security policies of the Pod. Here we add three strategies: host PID: true, host IPC: true, host network: true, which are used to use the PID of the host. Espace, IPC namespace and host network are key technologies for container isolation. Note that namespace and namespace in cluster are two completely different concepts.
Because the host network = true is specified, a port 9100 is bound to each node, through which we can obtain monitoring metrics data.
$ curl 127.0.0.1:9100/metrics
curl 127.0.0.1:10255/metrics# Monitors kubelet through 10255 port
2. Increase job and update prometheus configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: kube-ops
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      scrape_timeout: 15s
    scrape_configs:
    - job_name: 'prometheus'
      static_configs:
      - targets: ['localhost:9090']
    - job_name: 'traefik'
      static_configs:
      - targets: ['traefik-ingress-service.kube-system.svc.cluster.local:8080']
    - job_name: 'redis'
      static_configs:
      - targets: ['redis:9121']
    - job_name: 'kubernetes-nodes'
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - source_labels: [__address__]
        regex: '(.*):10250'
        replacement: '${1}:9100'
        target_label: __address__
        action: replace
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
    - job_name: 'kubernetes-kubelet'
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - source_labels: [__address__]
        regex: '(.*):10250'
        replacement: '${1}:10255'
        target_label: __address__
        action: replace
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)

Under Kubernetes, Promethues integrates with the Kubernetes API and currently supports five service discovery modes: Node, Service, Pod, Endpoints and Ingress.
The default listening ports of kubelet are 10250, 10255 and 10248, respectively.
$ vim /var/lib/kubelet/config.yaml
healthzPort: 10248
port: 10250
readOnlyPort: 10255
When prometheus went to discover Node mode service, the port of access defaulted to 10250, but now there is no / metrics index data under the port. Now the kubelet read-only data interface is exposed through 10255 port, so we should replace the port here, but we want to replace it. Is it port 10255? No, because we're going to configure the node metrics captured by node-exporter above, and do we specify hostNetwork=true above, so we'll bind a port 9100 on each node, so we should replace 10250 with 9100 here.
$ kubectl apply -f prometheus-cm.yaml
$ kubectl get svc -n kube-ops |grep prometheus
prometheus NodePort 10.102.197.83 <none> 9090:32619/TCP
$ curl -X POST "http://10.102.197.83:9090/-/reload "# Make the configuration effective