I'm having an issue with the Kubernetes provider. We are passing an explicit kubeconfig into the provider, containing certificate information. However, the certificates will expire soon, so we need to replace them. However, changing the

kubeconfig

argument to the provider causes the provider to be replaced, which also causes all resources in the stack to be replaced! We've worked around this by doing some stack editing, and manually replacing the

kubeconfig

input in the stack state, so the change is not detected. This doesn't seem like a sustainable workflow though, what is the recommended way of rotating provider credentials?

I don't have a solution but I'd be keen on learning how others deal with this as well

unfortunately it looks like you’re running into https://github.com/pulumi/pulumi-kubernetes/issues/2370 we’ve recently been discussing some similar issues and potential fixes. we’ll keep you posted.

Thanks for the link. Unfortunately, I don't think any of the workarounds are feasible (I've left a comment on the issue explaining), so I'm very interested in a better solution here.

That issue in https://github.com/pulumi/pulumi-kubernetes/issues/2370 - has anyone found a reasonable workaround? Reviving this thread just for today 🌇

Perhaps we could use an environment variable to cause the Pulumi provider to skip the cluster replacement check. Would this be an acceptable workaround @dazzling-oxygen-84405 and @salmon-gold-74709?

I think https://github.com/pulumi/pulumi-kubernetes/issues/2745 (which I see you've commented on already) would be even better, as it would make it transparent in the code. Not sure if that's easier or more difficult to implement than an environment variable. One of the projects I'm working on would benefit from being able to swap the kubeconfig (or alternate between two Kubernetes providers) based on the environment the Pulumi code is executed in. So we essentially have Kai's situation regularly.

Yeah, the linked proposal (if passing

clusterId

, only treat the provider as changed if this value changes) sounds like it would work for us. An env var would probably also be OK, but the

clusterId

seems more explicit. With the env var, I think it would have to be a "sticky" change, so once I run

PULUMI_K8S_NO_REPLACE_PROVIDER=1 pulumi up

, the provider remembers the updated kubeconfig, and the env var no longer needs to be set (until the kubeconfig changes again).

We are currently facing the same issues since we need to rotate our kubeconfigs for several clusters. The proposed

clusterId

sounds good as a solution for this

The clusterId proposal is the best option - easy to forget to set an env var, so if that’s done it should be sticky as @dazzling-oxygen-84405 said. This is really a severe issue - if Pulumi ever removed all resources this would be a major outage and reason to drop Pulumi for many

totally agree! we’ll be prioritizing the clusterid fix for #2745