I’m coming back to some multistack management questions, it keeps biting me in the ass so I think i’m doing something wrong: Stack 1: deploys a k8s cluster (gke/eks/aks) -> exports kubeconfig Stack 2: imports the kubeconfig from stack 1 -> makes a provider -> deploys stuff If i change anything in stack 1 that causes the cluster to be recreated (exact case this time, changed a gke clusters amount of nodes)and pressed yes too quickly because it was part of a large changeset) the cluster redeploys, the old one is deleted and the kubeconfig is updated. Any actions in stack 2 fail horribly. Because even though the kubeconfig is replaced, all the deployed k8s resources are trying to work on the old clusters ip address. It is like they store that in their state and not use a refreshed value. Or is it the provider that holds that link, and would exporting the provider be better? As the stacks are quite large on their own, manually the state file is next to impossible to get right, even with scripts. At this point the only way I know I can recover, is delete the stack completely and clean up on the cloud provider myself. Feels like a common thing to happen, so I assume I’m doing it wrong. Any pointers?

I’ve run into this as well (and makes me be concerned about changing anything that would cause the EKS cluster to be re-created).

I should “protect” the cluster when it is final, that would prevent anything bad to happen to it, but at this point tuning it is part of the job to be done.

Yes. In a way I would want to be able to say something like “for this provider, if the provider definition changed, just forget everything from the previous version, and just recreate it at all”. Currently the only way to do that is to destroy the entire stack basically.

Yeah, that is a bit hard since i have several clusters in different providers in one stack, i probably going to need a single one per cluster.

do you reload your kubernetes resource provider with an import from EKS and GenerateKubeconfig from the cluster stack to the kubernetes stack, I have also found that EnableDryRun = true is quite helpful as well as running a pulumi refresh to rebuild whats actually in the cluster before kicking off on a new cluster

that would not work as to get the refresh info, it needs to connect to the cluster, but it uses the old kubeconfig

Forgive me if this makes no sense - I use ECS so not 100% familiar with k8s terms / Config Would an auto scaling group help? I think for K8s it’s called a “cluster auto scaler” but would that provide an additional layer of abstraction on your cluster potentially handling that pass through of new Config when you need to make scaling changes on the ASG?

It is a bit different, i’m talking pulumi stacks. While you an get information (e.g. kubeconfig) from one stack, and use it to deploy something on that cluster, that stack is oblivious to anything that happens in that base stack. So if the kubeconfig changes, the stack using the base config does not realise this and tries to connect with the old kubeconfig.

Do you need to output the kubeconfig itself? Could you instead output some identifier for the cluster and have stack 2 fetch the kubeconfig on execution?