Hello. Have anyone experienced any issues with del...
# kubernetesw
wet-noon-14291
03/23/2021, 3:56 PMHello. Have anyone experienced any issues with deleting kubernetes namespaces? When I remove a namespace that has been provisioned by pulumi it ends up in a "terminating" state, like described here: https://stackoverflow.com/questions/52369247/namespace-stuck-as-terminating-how-do-i-remove-it#:~:text=This%20is%20caused%20by%20resources,controller%20is%20unable%20to%20remove.&text=You%20can%20edit%20namespace%20on,enter%20or%20save%2Fapply%20changes.
g
gentle-diamond-70147
03/23/2021, 4:05 PMHow did you remove the namespace?
b
billowy-army-68599
03/23/2021, 5:07 PM
are there any custom resources in that namespace?
w
wet-noon-14291
03/23/2021, 5:46 PM@billowy-army-68599 , no custom resources.
b
billowy-army-68599
03/23/2021, 5:47 PM
any finalizers on the namespace?
w
wet-noon-14291
03/23/2021, 5:47 PM@gentle-diamond-70147 , first run I created a service account and namespace, second run I wanted to remove them and then it got stuck.
g
gentle-diamond-70147
03/23/2021, 5:47 PMYou removed them with Pulumi or something else?
w
wet-noon-14291
03/23/2021, 5:48 PM@billowy-army-68599 , not anyone that I’ve added. But that is what “fixes” it. Add the config for the namespace and remove the finalizer. It is the “kubernetes” finalizer just likes described in the SO post.
wet-noon-14291
03/23/2021, 5:49 PM@gentle-diamond-70147, everything was done by pulumi as part of an automation.
wet-noon-14291
03/23/2021, 5:50 PMTo remove the namespace with kubectl I have to do what is mentioned in the SO post and remove the finalizer. I don’t even know how the finalizer got there or what it does 🙃
b
billowy-army-68599
03/23/2021, 5:50 PM
are you sure there isn't an operator (ie something outside of pulumi) creating something in the namespace?
billowy-army-68599
03/23/2021, 5:50 PM
the kubernetes finalizer is the default: https://book.kubebuilder.io/reference/using-finalizers.html
w
wet-noon-14291
03/23/2021, 5:51 PMIt is a pretty clean cluster that I’ve created for demo purposes. So nothing else. Could it be that when I add a service account it also adds a secret in the namespace?
b
billowy-army-68599
03/23/2021, 5:52 PM
the sa does have a default secret which is the token, but that shouldn't be the thing
billowy-army-68599
03/23/2021, 5:52 PM
is it EKS? generally you have to look at the controller logs to figure this out
w
wet-noon-14291
03/23/2021, 5:52 PMThe service account was also added with pulumi, so I hope pulumi will clean the secret app when the service account is deleted...
wet-noon-14291
03/23/2021, 5:52 PMIt’s AKS
wet-noon-14291
03/23/2021, 5:53 PMNot at the computer right now, but if I get some pointers to what I should check I can try to do it when I have time. Hopefully later today.
wet-noon-14291
03/23/2021, 5:53 PMI’ll also try to create a smaller reproduction if possible.
b
billowy-army-68599
03/23/2021, 5:54 PM
try this
Copy code
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found <namespace>👍 1
w
wet-noon-14291
03/23/2021, 8:44 PM@billowy-army-68599, the error I see is
error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
wet-noon-14291
03/23/2021, 8:46 PM(I think your command was slightly wrong, was missing
-n<namespace>b
billowy-army-68599
03/23/2021, 10:11 PM
Ah yes sorry, copy paste error
w
wet-noon-14291
03/24/2021, 8:11 AMI wonder if I have not enough resources that is causing issues.
wet-noon-14291
03/24/2021, 9:09 PMI'm pretty sure that it had to do with not enough resources. Since I did create the cluster with some metrics and other stuff, two nodes might not have been enough. Creating the cluster again with three nodes and I have no problem with deleting a namespace.
4 Views