https://pulumi.com logo
#kubernetes
Title
# kubernetes
a

ancient-megabyte-79588

01/26/2021, 6:57 PM
Hello k8s peeps... I'm not sure how this has happened, but my pulumi script for deploying apps into our cluster (nginx-ingress-controller, fluentd, certmanager, etc) has gotten into a condition where the stack thinks there should be some resources in the cluster that are simply not there and
pulumi refresh
doesn't work to bring the stack into sync with the cluster. The concerning thing is that these resources were removed by the pulumi script, but the script failed in a latter part of the execution (couldn't get ip address property from
ingressController
object), and so the stack is now out of sync with what the script removed, and I can't get the stack back in sync. I cannot
pulumi destroy
the stack since there are things in place that would be very problematic to re-create.
g

gentle-diamond-70147

01/26/2021, 6:59 PM
When you say pulumi refresh doesn't work, can you elaborate?
a

ancient-megabyte-79588

01/26/2021, 7:08 PM
as I understand it
pulumi refresh
inspects the stack and the script, and tries to bring them together. So if the app doesn't have a resource described, and the cluster doesn't have it, but the stack does have that resource, it would remove it from the stack.
As an example, I have removed certmanager from the script, and it was removed from the cluster, but the stack seems to still think that all of the certmanager resources are still present.
Currently the script does not have a cert-manager chart defined anywhere, the cluster does not have any cert-manager resources, so this picture of the stack and resources is incorrect and I would expect
pulumi refresh
to fix that.
So we have exported the stack to a .json file and we've manually removed all of the dangling resources and re-imported it and that seems to have worked.
g

gentle-diamond-70147

01/26/2021, 8:46 PM
Yes, I would have expected
pulumi refresh
to fix that too. To expand a bit on the behavior you described... if you have a resource in your Pulumi state, but that resource does not exist in your provider,
pulumi refresh
should "fix" that, meaning recognize the resource no longer exists and remove from your state file.
Having said that, I believe the
v3:Chart
resource itself is a component resource in Pulumi's terms, so it does not exist in that exact form in your k8s cluster. So
pulumi refresh
should have removed any of the sub-resources of that chart (if they no longer existed), but the chart component might not be removed. I'll need to confirm the "component refresh" behavior with a Pulumi engineer.
a

ancient-megabyte-79588

01/26/2021, 10:32 PM
Thanks @gentle-diamond-70147 for looking into it. I recently updated to the latest CLI so I was a bit concerned a regression had been introduced.
I've never had to manually manipulate the export of a stack to get to "clean".
4 Views