TLDR; DigitalOcean rotates the kubeconfig, and eve...
# kubernetesm
mammoth-afternoon-82670
09/30/2020, 6:56 PMTLDR; DigitalOcean rotates the kubeconfig, and everytime it does that, pulumi is unable to connect to the cluster due to server asking for credentials. I tried doing a refresh with
--targetdigitalocean.KubernetesClusterconfigured Kubernetes cluster is unreachable: unable to load schema information from the API server: the server has asked for the client to provide credentialsg
gorgeous-egg-16927
09/30/2020, 8:07 PMYou should be able to revert to any previous checkpoint state with the following:
pulumi stack export --version=<previous-version-number> > outpulumi stack import --file=outm
mammoth-afternoon-82670
09/30/2020, 8:12 PMI did revert, but I'm unsure how to update the kubeconfig now
mammoth-afternoon-82670
09/30/2020, 8:13 PMI did a
pulumi refresh --target urn:clusterpulumi refreshmammoth-afternoon-82670
09/30/2020, 8:20 PMI imagine this is the part that is supposed to keep the
kubeconfigpulumi.log.errorg
gorgeous-egg-16927
09/30/2020, 8:49 PMHmm…I haven’t used DO k8s much, but I you should be able to temporarily set the kubeconfig as a stack config to get yourself unstuck. (
pulumi config set kubeconfig ...gorgeous-egg-16927
09/30/2020, 8:49 PMI thought I recall somebody saying that you can get longer-term kubeconfigs for automation scenarios to help prevent this in the future, but I’m not 100% sure on that
m
mammoth-afternoon-82670
09/30/2020, 9:25 PMI'm spawning a new cluster with k3d locally to see if I can reproduce the problem
mammoth-afternoon-82670
09/30/2020, 9:26 PMI don't know how pulumi handles resource relationship input/output resolution (pull or push)
mammoth-afternoon-82670
09/30/2020, 9:27 PMso I don't really know if it's DO resource or K8s one's failing to update the information in the state
mammoth-afternoon-82670
09/30/2020, 9:33 PMI think I saw the workaround you mentioned, but I'm unable to find and test it at this point.
g
gorgeous-egg-16927
09/30/2020, 9:41 PMI think this is the issue: https://github.com/pulumi/pulumi-digitalocean/issues/78
m
mammoth-afternoon-82670
09/30/2020, 9:42 PMyeah, just found it too. I'll test if this approach works
mammoth-afternoon-82670
09/30/2020, 9:42 PMIt's the second time this happens, the last time I had to delete the whole cluster because I couldn't recover the state it was before
mammoth-afternoon-82670
09/30/2020, 9:43 PMI think I`m better off splitting the infrastructure from the application. This is causing a lot of problems and I won't be able to publish this to production since this is managing enterprise client clusters 😐
mammoth-afternoon-82670
09/30/2020, 9:45 PMI'd love to track this down and make a reproducible demo, but I'm unsure how to properly debug what is going on (doesn't log, doesn't trace, process.exit seems to be ignored, throw doesn't do a thing, enabling --debug generates a 25mb output)
mammoth-afternoon-82670
09/30/2020, 9:47 PMonly thing I can imagine is that refresh differs from up on the code execution somehow? but I have no idea how it's supposed to refresh stuff without executing the code and
apply()g
gorgeous-egg-16927
09/30/2020, 9:56 PMSorry you’re having a bad experience here. I would recommend trying to keep cluster infra in a separate stack from k8s applications, as it helps limit the “blast radius” when something changes. We do have logic in the provider to handle unreachable clusters, but it seems like that’s not working right in your case.
Refresh/diff is a separate code path from update, but I’m not sure if that’s causing the behavior you’re reporting.
m
mammoth-afternoon-82670
09/30/2020, 9:59 PMIt's fine, I'm pro in finding edge cases hahaha, I'll keep digging later until I get this working
mammoth-afternoon-82670
09/30/2020, 9:59 PMHopefully I can find what is causing it
6 Views