Hello k8s peeps I m not sure how this has happened but my pu Pulumi Community #kubernetes

Hello k8s peeps... I'm not sure how this has happ...

ancient-megabyte-79588

01/26/2021, 6:57 PM

Hello k8s peeps... I'm not sure how this has happened, but my pulumi script for deploying apps into our cluster (nginx-ingress-controller, fluentd, certmanager, etc) has gotten into a condition where the stack thinks there should be some resources in the cluster that are simply not there and

pulumi refresh

doesn't work to bring the stack into sync with the cluster. The concerning thing is that these resources were removed by the pulumi script, but the script failed in a latter part of the execution (couldn't get ip address property from

ingressController

object), and so the stack is now out of sync with what the script removed, and I can't get the stack back in sync. I cannot

pulumi destroy

the stack since there are things in place that would be very problematic to re-create.

gentle-diamond-70147

01/26/2021, 6:59 PM

When you say pulumi refresh doesn't work, can you elaborate?

ancient-megabyte-79588

01/26/2021, 7:08 PM

as I understand it

pulumi refresh

inspects the stack and the script, and tries to bring them together. So if the app doesn't have a resource described, and the cluster doesn't have it, but the stack does have that resource, it would remove it from the stack.

ancient-megabyte-79588

01/26/2021, 7:09 PM

As an example, I have removed certmanager from the script, and it was removed from the cluster, but the stack seems to still think that all of the certmanager resources are still present.

ancient-megabyte-79588

01/26/2021, 7:12 PM

Currently the script does not have a cert-manager chart defined anywhere, the cluster does not have any cert-manager resources, so this picture of the stack and resources is incorrect and I would expect

pulumi refresh

to fix that.

ancient-megabyte-79588

01/26/2021, 7:38 PM

So we have exported the stack to a .json file and we've manually removed all of the dangling resources and re-imported it and that seems to have worked.

gentle-diamond-70147

01/26/2021, 8:46 PM

Yes, I would have expected

pulumi refresh

to fix that too. To expand a bit on the behavior you described... if you have a resource in your Pulumi state, but that resource does not exist in your provider,

pulumi refresh

should "fix" that, meaning recognize the resource no longer exists and remove from your state file.

gentle-diamond-70147

01/26/2021, 8:48 PM

Having said that, I believe the

v3:Chart

resource itself is a component resource in Pulumi's terms, so it does not exist in that exact form in your k8s cluster. So

pulumi refresh

should have removed any of the sub-resources of that chart (if they no longer existed), but the chart component might not be removed. I'll need to confirm the "component refresh" behavior with a Pulumi engineer.

ancient-megabyte-79588

01/26/2021, 10:32 PM

Thanks @gentle-diamond-70147 for looking into it. I recently updated to the latest CLI so I was a bit concerned a regression had been introduced.

ancient-megabyte-79588

01/26/2021, 10:32 PM

I've never had to manually manipulate the export of a stack to get to "clean".

4 Views

Open in Slack

Previous Next