Author: SRE operation and maintenance blog
Blog address: https://www.cnsre.cn/
Article address: https://www.cnsre.cn/posts/220107906441/
Related topics: https://www.cnsre.cn/tags/eks/
introduce
Recently, I have been playing EKS (Elastic Kubernetes Service -- Amazon EKS) and kubesphere. Because EKS and kubesphere have not been used before, this process is also a trial and error process. When I used kubesphere, I found a log service. Driven by curiosity, I created it.
After I created the KubeSphere Logging System, I found that I didn't want to use it. (maybe I just want to see what it is.) OCD is mine. I just want to delete it. So I forced him to delete kubectl delete ns KubeSphere Logging System -- force -- grace period = 0 in my EKS
It's embarrassing that the Namespace was not deleted immediately. I comforted myself that there may be other resources under the Namespace that are not deleted waiting to be deleted. I'm waiting...
After half an hour, when I went to check
[root@ip-10-0-100-206 ~]# kubectl get ns kubesphere-logging-system NAME STATUS AGE kubesphere-logging-system Terminating 6d19h
It seems to be stuck in the Terminating state.
I tried to find a solution, http://github.com/kubernetes/... However, this method can only be implemented through API. eks is hosted in AWS. I have no way to operate the background of eks.
I began to regret why I installed it. I couldn't have bothered. But looking back, if I didn't install it, I would be so sad as I thought, because I couldn't understand what the KubeSphere Logging System was like.
Finally I found commented https://github.com/kubernetes...
How to delete a namespace completely
Get the details of the namespace and turn it into json
kubectl get namespace kubesphere-logging-system -o json > kubesphere-logging-system.json
Open json file edit
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"creationTimestamp": "2021-12-31T05:03:58Z",
"deletionTimestamp": "2022-01-05T08:05:40Z",
"labels": {
"kubesphere.io/namespace": "kubesphere-logging-system",
"kubesphere.io/workspace": "system-workspace"
},
"managedFields": [
{
"apiVersion": "v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:labels": {
".": {},
"f:kubesphere.io/namespace": {}
},
"f:ownerReferences": {
".": {},
"k:{\"uid\":\"6d535470-2592-4f3c-a155-eabc362c339d\"}": {
".": {},
"f:apiVersion": {},
"f:blockOwnerDeletion": {},
"f:controller": {},
"f:kind": {},
"f:name": {},
"f:uid": {}
}
}
}
},
"manager": "controller-manager",
"operation": "Update",
"time": "2021-12-31T05:04:01Z"
},
{
"apiVersion": "v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:labels": {
"f:kubesphere.io/workspace": {}
}
},
"f:status": {
"f:phase": {}
}
},
"manager": "kubectl",
"operation": "Update",
"time": "2021-12-31T05:04:01Z"
},
{
"apiVersion": "v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:conditions": {
".": {},
"k:{\"type\":\"NamespaceContentRemaining\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceDeletionContentFailure\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceFinalizersRemaining\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
}
}
},
"manager": "kube-controller-manager",
"operation": "Update",
"time": "2022-01-05T08:05:47Z"
}
],
"name": "kubesphere-logging-system",
"ownerReferences": [
{
"apiVersion": "tenant.kubesphere.io/v1alpha1",
"blockOwnerDeletion": true,
"controller": true,
"kind": "Workspace",
"name": "system-workspace",
"uid": "6d535470-2592-4f3c-a155-eabc362c339d"
}
],
"resourceVersion": "7376520",
"uid": "2b76e9b1-75f2-4a2e-a819-73b36aea188e"
},
"spec": {
"finalizers": [
"kubernetes" # Delete this row
]
},
"status": {
"conditions": [
{
"lastTransitionTime": "2022-01-05T08:05:47Z",
"message": "All resources successfully discovered",
"reason": "ResourcesDiscovered",
"status": "False",
"type": "NamespaceDeletionDiscoveryFailure"
},
{
"lastTransitionTime": "2022-01-05T08:05:47Z",
"message": "All legacy kube types successfully parsed",
"reason": "ParsedGroupVersions",
"status": "False",
"type": "NamespaceDeletionGroupVersionParsingFailure"
},
{
"lastTransitionTime": "2022-01-05T08:05:47Z",
"message": "All content successfully deleted, may be waiting on finalization",
"reason": "ContentDeleted",
"status": "False",
"type": "NamespaceDeletionContentFailure"
},
{
"lastTransitionTime": "2022-01-05T08:05:47Z",
"message": "Some resources are remaining: fluentbits.logging.kubesphere.io has 1 resource instances",
"reason": "SomeResourcesRemain",
"status": "True",
"type": "NamespaceContentRemaining"
},
{
"lastTransitionTime": "2022-01-05T08:05:47Z",
"message": "Some content in the namespace has finalizers remaining: fluentbit.logging.kubesphere.io in 1 resource instances",
"reason": "SomeFinalizersRemain",
"status": "True",
"type": "NamespaceFinalizersRemaining"
}
],
"phase": "Terminating"
}
}Find the spec and delete the kubernetes under finalizers
The details are as follows
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"creationTimestamp": "2021-12-31T05:03:58Z",
"deletionTimestamp": "2022-01-05T08:05:40Z",
"labels": {
"kubesphere.io/namespace": "kubesphere-logging-system",
"kubesphere.io/workspace": "system-workspace"
},
"managedFields": [
{
"apiVersion": "v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:labels": {
".": {},
"f:kubesphere.io/namespace": {}
},
"f:ownerReferences": {
".": {},
"k:{\"uid\":\"6d535470-2592-4f3c-a155-eabc362c339d\"}": {
".": {},
"f:apiVersion": {},
"f:blockOwnerDeletion": {},
"f:controller": {},
"f:kind": {},
"f:name": {},
"f:uid": {}
}
}
}
},
"manager": "controller-manager",
"operation": "Update",
"time": "2021-12-31T05:04:01Z"
},
{
"apiVersion": "v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:labels": {
"f:kubesphere.io/workspace": {}
}
},
"f:status": {
"f:phase": {}
}
},
"manager": "kubectl",
"operation": "Update",
"time": "2021-12-31T05:04:01Z"
},
{
"apiVersion": "v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:conditions": {
".": {},
"k:{\"type\":\"NamespaceContentRemaining\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceDeletionContentFailure\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceFinalizersRemaining\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
}
}
},
"manager": "kube-controller-manager",
"operation": "Update",
"time": "2022-01-05T08:05:47Z"
}
],
"name": "kubesphere-logging-system",
"ownerReferences": [
{
"apiVersion": "tenant.kubesphere.io/v1alpha1",
"blockOwnerDeletion": true,
"controller": true,
"kind": "Workspace",
"name": "system-workspace",
"uid": "6d535470-2592-4f3c-a155-eabc362c339d"
}
],
"resourceVersion": "7376520",
"uid": "2b76e9b1-75f2-4a2e-a819-73b36aea188e"
},
"spec": {
"finalizers": [
]
},
"status": {
"conditions": [
{
"lastTransitionTime": "2022-01-05T08:05:47Z",
"message": "All resources successfully discovered",
"reason": "ResourcesDiscovered",
"status": "False",
"type": "NamespaceDeletionDiscoveryFailure"
},
{
"lastTransitionTime": "2022-01-05T08:05:47Z",
"message": "All legacy kube types successfully parsed",
"reason": "ParsedGroupVersions",
"status": "False",
"type": "NamespaceDeletionGroupVersionParsingFailure"
},
{
"lastTransitionTime": "2022-01-05T08:05:47Z",
"message": "All content successfully deleted, may be waiting on finalization",
"reason": "ContentDeleted",
"status": "False",
"type": "NamespaceDeletionContentFailure"
},
{
"lastTransitionTime": "2022-01-05T08:05:47Z",
"message": "Some resources are remaining: fluentbits.logging.kubesphere.io has 1 resource instances",
"reason": "SomeResourcesRemain",
"status": "True",
"type": "NamespaceContentRemaining"
},
{
"lastTransitionTime": "2022-01-05T08:05:47Z",
"message": "Some content in the namespace has finalizers remaining: fluentbit.logging.kubesphere.io in 1 resource instances",
"reason": "SomeFinalizersRemain",
"status": "True",
"type": "NamespaceFinalizersRemaining"
}
],
"phase": "Terminating"
}
}Execute cleanup command
Now we only need one command to completely delete the Namespace
kubectl replace --raw "/api/v1/namespaces/kubesphere-logging-system/finalize" -f ./kubesphere-logging-system.json
After execution, you need to wait for a while
Execute the command again to check the namespaces
kubectl replace --raw "/api/v1/namespaces/kubesphere-logging-system/finalize" -f ./kubesphere-logging-system.json
Final inspection
kubectl get ns kubesphere-logging-system Error from server (NotFound): namespaces "kubesphere-logging-system" not found [root@ip-10-0-100-206 ~]# kubectl get ns NAME STATUS AGE default Active 23d kubesphere-controls-system Active 9d kubesphere-devops-system Active 9d kubesphere-devops-worker Active 16h kubesphere-monitoring-federated Active 9d kubesphere-monitoring-system Active 9d kubesphere-sample-dev Active 8d kubesphere-system Active 9d
When viewed here, it no longer exists.
Author: SRE operation and maintenance blog
Blog address: https://www.cnsre.cn/
Article address: https://www.cnsre.cn/posts/220107906441/
Related topics: https://www.cnsre.cn/tags/eks/