phenomenon
An internal error occurred on the harbor mirror platform server and the request could not be completed
Investigation
Logging on to dashboard finds that the node state is not ready, and then logs on to the node
When you see that they are indeed not ready, your instinct is to rerun the kubelet, kube-proxy services and return them to the notread state.
A closer look at the nodes node status age of 368d at this point indicates what should be expired. The certificate was deliberately applied for for 10 years before, which should not be a certificate issue. Later, consulting the deployment documentation, it was found that kubelet also needs certificate authorization.
Solve
Finally, kubectl get csr found that, as a result, node-csr is in Pending state, it needs to re-kubectl certificate approve to take effect.
Looking up the data, we found that the default signed certificate only has one year validity period. If you want to adjust the validity period of the certificate, you can do so by setting the--experimental-cluster-signing-duration parameter of kube-controller-manager, which has a default value of 8760 h0m0s. You can refer to how to increase the default signing time and automatic certificate rotation. kubernetes authentication and authorization mechanism.
step
[root@k8s01 bin]# kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.88.34 NotReady <none> 368d v1.13.1 192.168.88.65 NotReady <none> 368d v1.13.1 [root@k8s01 bin]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-d8End93rQqSFRHpV65KA2yL-4CnYHT4te6D85lzO5QA 18m kubelet-bootstrap Pending node-csr-iuBw7qxuiCeyiQ1x5WqTASgqVheII-KpTX0L-S8Md8Y 10m kubelet-bootstrap Pending [root@k8s01 bin]# kubectl certificate approve node-csr-d8End93rQqSFRHpV65KA2yL-4CnYHT4te6D85lzO5QA certificatesigningrequest.certificates.k8s.io/node-csr-d8End93rQqSFRHpV65KA2yL-4CnYHT4te6D85lzO5QA approved [root@k8s01 bin]# kubectl certificate approve node-csr-iuBw7qxuiCeyiQ1x5WqTASgqVheII-KpTX0L-S8Md8Y certificatesigningrequest.certificates.k8s.io/node-csr-iuBw7qxuiCeyiQ1x5WqTASgqVheII-KpTX0L-S8Md8Y approved [root@k8s01 bin]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-d8End93rQqSFRHpV65KA2yL-4CnYHT4te6D85lzO5QA 19m kubelet-bootstrap Approved,Issued node-csr-iuBw7qxuiCeyiQ1x5WqTASgqVheII-KpTX0L-S8Md8Y 11m kubelet-bootstrap Approved,Issued [root@k8s01 bin]# kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.88.34 Ready <none> 368d v1.13.1 192.168.88.65 Ready <none> 368d v1.13.1