This message was deleted.
# generals
sparse-intern-71089
09/02/2021, 6:42 AMThis message was deleted.
b
bored-table-20691
09/02/2021, 6:44 AMI’ve run into this as well (and makes me be concerned about changing anything that would cause the EKS cluster to be re-created).
bored-table-20691
09/02/2021, 6:44 AMI think part of the challenge is that if you change the Kubernetes link (e.g. kubeconfig changes), then the state thinks that it needs to delete it from the old cluster, which is obviously not available anymore, so that fails.
bored-table-20691
09/02/2021, 6:45 AMIt would happen with similar things (e.g. you create RDS in one project/stack and DBs in that RDS in another - if you recreate the RDS, you will be somewhat hosed when you try and update the dependent stack)
p
proud-pizza-80589
09/02/2021, 6:50 AMI should “protect” the cluster when it is final, that would prevent anything bad to happen to it, but at this point tuning it is part of the job to be done.
b
bored-table-20691
09/02/2021, 7:00 AMYes. In a way I would want to be able to say something like “for this provider, if the provider definition changed, just forget everything from the previous version, and just recreate it at all”.
Currently the only way to do that is to destroy the entire stack basically.
p
proud-pizza-80589
09/02/2021, 7:22 AMYeah, that is a bit hard since i have several clusters in different providers in one stack, i probably going to need a single one per cluster.
s
salmon-guitar-84472
09/02/2021, 9:02 AMdo you reload your kubernetes resource provider with an import from EKS and GenerateKubeconfig from the cluster stack to the kubernetes stack, I have also found that EnableDryRun = true is quite helpful as well as running a pulumi refresh to rebuild whats actually in the cluster before kicking off on a new cluster
p
proud-pizza-80589
09/02/2021, 11:17 AMthat would not work as to get the refresh info, it needs to connect to the cluster, but it uses the old kubeconfig
b
bored-oyster-3147
09/02/2021, 11:36 AMForgive me if this makes no sense - I use ECS so not 100% familiar with k8s terms / Config
Would an auto scaling group help? I think for K8s it’s called a “cluster auto scaler” but would that provide an additional layer of abstraction on your cluster potentially handling that pass through of new Config when you need to make scaling changes on the ASG?
bored-oyster-3147
09/02/2021, 11:37 AMBut yea I would def recommend marking the problem clusters as protected in the mean time, if only so you can coordinate changes / downtime and are no longer surprised
bored-oyster-3147
09/02/2021, 11:46 AMI’m almost more surprised that the cloud provider even lets you delete the cluster when you still have nodes referencing that Config. I must be misunderstanding something about k8s there
p
proud-pizza-80589
09/02/2021, 1:13 PMIt is a bit different, i’m talking pulumi stacks. While you an get information (e.g. kubeconfig) from one stack, and use it to deploy something on that cluster, that stack is oblivious to anything that happens in that base stack. So if the kubeconfig changes, the stack using the base config does not realise this and tries to connect with the old kubeconfig.
proud-pizza-80589
09/02/2021, 1:14 PMNow if you change something that replaces the entire cluster, then that kubeconfig of the old cluster does not work (obviously)
proud-pizza-80589
09/02/2021, 1:16 PMA solution would be:
• the importing stack checks if the output it used is the same. It could then just use the new value, and combined with --refresh it would notice that the pods are gone and redeploy them.
• or alternatively, allow for destroy --do-your-best-and-ignore-leftovers, which you could combine with --delete-failures-from-the-state
proud-pizza-80589
09/02/2021, 1:17 PMOption 1 is very k8s specific since in other infra this is not necessarily the way to go
proud-pizza-80589
09/02/2021, 1:18 PMoption 2 could leave leftovers, and in general ususally all the tools prefer to be very strict on you cleaning up yourself.
b
bored-oyster-3147
09/02/2021, 2:05 PMDo you need to output the kubeconfig itself? Could you instead output some identifier for the cluster and have stack 2 fetch the kubeconfig on execution?
4 Views