Problems encountered in deployment of kubernetes 1.13.3 heapster

Background: Recently, I saw the k8s tutorial of Marco. In the chapter of container resource requirements, resource limitations and headstar, the kubectl top and grafana figures in the tutorial were not shown at last. Heapster will be discarded in later versions, so there is no need to struggle. I'm just curious. Next, I will talk about the problems and solutions. The version of k8s I installed is v1.13.3.

View version

[ryuser@cdh-master metrics]$ kubectl get nodes
NAME                    STATUS   ROLES    AGE   VERSION   Ready    master   41d   v1.13.3    Ready    <none>   41d   v1.13.3   Ready    <none>   39d   v1.13.3


1. When creating a heapster, the following error always occurs when viewing the log

[ryuser@cdh-master metrics]$ kubectl logs heapster-f64999bc-25tvv -n kube-system
I0326 06:23:03.317063       1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
I0326 06:23:03.317170       1 heapster.go:79] Heapster version v1.5.4
I0326 06:23:03.317421       1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0326 06:23:03.317437       1 configs.go:62] Using kubelet port 10255
I0326 06:23:03.341940       1 influxdb.go:312] created influxdb sink with options: host:monitoring-influxdb.kube-system.svc:8086 user:root db:k8s
I0326 06:23:03.341976       1 heapster.go:202] Starting with InfluxDB Sink
I0326 06:23:03.341985       1 heapster.go:202] Starting with Metric Sink
I0326 06:23:03.364225       1 heapster.go:112] Starting heapster on port 8082
E0326 06:24:05.006245       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": Post dial tcp getsockopt: connection refused
E0326 06:24:05.006326       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": Post dial tcp getsockopt: connection refused
E0326 06:24:05.006827       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": Post dial tcp getsockopt: connection refused
W0326 06:24:25.002576       1 manager.go:152] Failed to get all responses in time (got 0/3)
I0326 06:24:25.033246       1 influxdb.go:274] Created database "k8s" on influxDB server at "monitoring-influxdb.kube-system.svc:8086"
E0326 06:25:05.009902       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": Post dial tcp getsockopt: connection refused
E0326 06:25:05.010317       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": Post dial tcp getsockopt: connection refused
E0326 06:25:05.024937       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": Post dial tcp getsockopt: connection refused
W0326 06:25:25.002198       1 manager.go:152] Failed to get all responses in time (got 0/3)
E0326 06:26:05.011184       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": Post dial tcp getsockopt: connection refused
E0326 06:26:05.014660       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": Post dial tcp getsockopt: connection refused
E0326 06:26:05.021066       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": Post dial tcp getsockopt: connection refused


2. The kubectl top command can't get the desired result

[ryuser@cdh-master metrics]$ kubectl top pod
W0326 15:13:19.303263 20846 top_pod.go:259] Metrics not available for pod default/client, age: 980h4m21.303224766s
error: Metrics not available for pod default/client, age: 980h4m21.303224766s
[ryuser@cdh-master metrics]$ kubectl top node
error: metrics not available yet

terms of settlement:

#Make the following changes in the heapster.yaml manifest file
- --source=kubernetes:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true 
- --sink=influxdb:http://monitoring-influxdb.kube-system.svc.cluster.local:8086


Then delete the heapster rebuild

kubectl delete -f heapster.yaml

kubectl apply -f heapster.yaml




3. Another 403 error


403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"

[ryuser@cdh-master metrics]$ kubectl logs -f heapster-5fcf457b-zq99c  -n kube-system
I0326 07:36:23.229287       1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true --sink=influxdb:http://monitoring-influxdb.kube-system.svc.cluster.local:8086
I0326 07:36:23.229348       1 heapster.go:79] Heapster version v1.5.4
I0326 07:36:23.229602       1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0326 07:36:23.229618       1 configs.go:62] Using kubelet port 10250
I0326 07:36:23.334904       1 influxdb.go:312] created influxdb sink with options: host:monitoring-influxdb.kube-system.svc.cluster.local:8086 user:root db:k8s
I0326 07:36:23.334946       1 heapster.go:202] Starting with InfluxDB Sink
I0326 07:36:23.334955       1 heapster.go:202] Starting with Metric Sink
I0326 07:36:23.347573       1 heapster.go:112] Starting heapster on port 8082
E0326 07:37:05.028341       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
E0326 07:37:05.096629       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
E0326 07:37:05.157683       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
W0326 07:37:25.003226       1 manager.go:152] Failed to get all responses in time (got 0/3)
I0326 07:37:25.037245       1 influxdb.go:274] Created database "k8s" on influxDB server at "monitoring-influxdb.kube-system.svc.cluster.local:8086"
E0326 07:38:05.013221       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
E0326 07:38:05.019540       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
E0326 07:38:05.022849       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
W0326 07:38:25.003081       1 manager.go:152] Failed to get all responses in time (got 0/3)
E0326 07:39:05.010246       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
E0326 07:39:05.019238       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
E0326 07:39:05.024794       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
W0326 07:39:25.004236       1 manager.go:152] Failed to get all responses in time (got 0/3)
E0326 07:40:05.016757       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
E0326 07:40:05.020030       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
E0326 07:40:05.020763       1 manager.go:101] Error in scraping containers from kubelet: failed to get all container stats from Kubelet URL "": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
W0326 07:40:25.002318       1 manager.go:152] Failed to get all responses in time (got 0/3)



terms of settlement:

Check the permissions of ClusterRole: system:heapster and find that there is no create permission for Resource: nodes/stats

[ryuser@cdh-master metrics]$ kubectl describe clusterrole system:heapster
Name:         system:heapster
  Resources               Non-Resource URLs  Resource Names  Verbs
  ---------               -----------------  --------------  -----
  events                  []                 []              [get list watch]
  namespaces              []                 []              [get list watch]
  nodes/stats             []                 []              [get list watch]
  nodes                   []                 []              [get list watch]
  pods                    []                 []              [get list watch]
  deployments.extensions  []                 []              [get list watch]

Modify the permissions of ClusterRole: system:heapster

Generate manifest file

kubectl get clusterrole system:heapster -o yaml > heapster_modify.yaml

Modify the file, increase the permissions of verbs:create, and resources:nodes/stats

vim  heapster_modify.yaml

kind: ClusterRole
  annotations: |
      {"apiVersion":"","kind":"ClusterRole","metadata":{"annotations":{"":"true"},"creationTimestamp":"2019-02-12T10:41:33Z","labels":{"":"rbac-defaults"},"name":"system:heapster","resourceVersion":"70","selfLink":"/apis/","uid":"c3bd303a-2eb2-11e9-9c98-005056be639a"},"rules":[{"apiGroups":[""],"resources":["events","namespaces","nodes","pods"],"verbs":["create","get","list","watch"]},{"apiGroups":["extensions"],"resources":["deployments"],"verbs":["get","list","watch"]}]} "true"
  creationTimestamp: "2019-02-12T10:41:33Z"
  labels: rbac-defaults
  name: system:heapster
  resourceVersion: "4109335"
  selfLink: /apis/
  uid: c3bd303a-2eb2-11e9-9c98-005056be639a
- apiGroups:
  - ""
  - events
  - namespaces
  - nodes
  - pods
  - nodes/stats  #Increase
  - create   #increase
  - get
  - list
  - watch
- apiGroups:
  - extensions
  - deployments
  - get
  - list
  - watch


Delete heapster redeployment

kubectl delete -f heapster.yaml

kubectl apply -f heapster.yaml



At last, I didn't make a mistake.

[ryuser@cdh-master metrics]$ kubectl logs -f heapster-5fcf457b-vhrxf  -n kube-system
I0326 07:47:00.574138       1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true --sink=influxdb:http://monitoring-influxdb.kube-system.svc.cluster.local:8086
I0326 07:47:00.574204       1 heapster.go:79] Heapster version v1.5.4
I0326 07:47:00.574470       1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0326 07:47:00.574487       1 configs.go:62] Using kubelet port 10250
I0326 07:47:00.639292       1 influxdb.go:312] created influxdb sink with options: host:monitoring-influxdb.kube-system.svc.cluster.local:8086 user:root db:k8s
I0326 07:47:00.639338       1 heapster.go:202] Starting with InfluxDB Sink
I0326 07:47:00.639354       1 heapster.go:202] Starting with Metric Sink
I0326 07:47:00.670576       1 heapster.go:112] Starting heapster on port 8082
I0326 07:48:05.366442       1 influxdb.go:274] Created database "k8s" on influxDB server at "monitoring-influxdb.kube-system.svc.cluster.local:8086"

kubectl top

[ryuser@cdh-master metrics]$ kubectl top nodes
NAME                    CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   158m         3%     2550Mi          69%    79m          1%     2386Mi          64%   820m         41%    3136Mi          84%       
[ryuser@cdh-master metrics]$ kubectl top pods
NAME                         CPU(cores)   MEMORY(bytes)   
curl-66959f6557-bvn9r        0m           0Mi             
dep-httpd-5b774f45df-vtv59   0m           21Mi            
dep-httpd-5b774f45df-wd5kf   0m           15Mi            
myapp-0                      0m           1Mi             
myapp-1                      0m           3Mi             
myapp-2                      0m           1Mi             
myapp-3                      0m           1Mi             
myapp-4                      0m           1Mi             
pod-demo                     499m         138Mi


Another problem is that the dashboard in grafana does not display data. After the above toss, we have data.


Attached: download address of dashboard:

"Kubernetes Node Statistics"dashabord :

"Kubernetes Pod Statistics"dashabord:

Keywords: Linux kubelet InfluxDB Kubernetes Database

Added by ksimpkins on Wed, 04 Dec 2019 00:50:07 +0200