I'm having an issue with the Kubernetes provider. ...
# kubernetes
d
I'm having an issue with the Kubernetes provider. We are passing an explicit kubeconfig into the provider, containing certificate information. However, the certificates will expire soon, so we need to replace them. However, changing the
kubeconfig
argument to the provider causes the provider to be replaced, which also causes all resources in the stack to be replaced! We've worked around this by doing some stack editing, and manually replacing the
kubeconfig
input in the stack state, so the change is not detected. This doesn't seem like a sustainable workflow though, what is the recommended way of rotating provider credentials?
m
I don't have a solution but I'd be keen on learning how others deal with this as well
h
unfortunately it looks like you’re running into https://github.com/pulumi/pulumi-kubernetes/issues/2370 we’ve recently been discussing some similar issues and potential fixes. we’ll keep you posted.
d
Thanks for the link. Unfortunately, I don't think any of the workarounds are feasible (I've left a comment on the issue explaining), so I'm very interested in a better solution here.
s
That issue in https://github.com/pulumi/pulumi-kubernetes/issues/2370 - has anyone found a reasonable workaround? Reviving this thread just for today 🌇
d
Perhaps we could use an environment variable to cause the Pulumi provider to skip the cluster replacement check. Would this be an acceptable workaround @dazzling-oxygen-84405 and @salmon-gold-74709?
m
I think https://github.com/pulumi/pulumi-kubernetes/issues/2745 (which I see you've commented on already) would be even better, as it would make it transparent in the code. Not sure if that's easier or more difficult to implement than an environment variable. One of the projects I'm working on would benefit from being able to swap the kubeconfig (or alternate between two Kubernetes providers) based on the environment the Pulumi code is executed in. So we essentially have Kai's situation regularly.
d
Yeah, the linked proposal (if passing
clusterId
, only treat the provider as changed if this value changes) sounds like it would work for us. An env var would probably also be OK, but the
clusterId
seems more explicit. With the env var, I think it would have to be a "sticky" change, so once I run
PULUMI_K8S_NO_REPLACE_PROVIDER=1 pulumi up
, the provider remembers the updated kubeconfig, and the env var no longer needs to be set (until the kubeconfig changes again).
m
We are currently facing the same issues since we need to rotate our kubeconfigs for several clusters. The proposed
clusterId
sounds good as a solution for this