https://pulumi.com logo
#kubernetes
Title
# kubernetes
f

full-boots-69133

09/11/2023, 11:55 PM
Hi all, does anyone here have any experience with dealing with differences between the Kubeconfigs between local and CI? The context for me is that Teleport's local k8s connections use
tsh
as the credential exec command and interacts with a user session (I check this is already logged in before running pulumi) while in CI we have to use
tbot
to interact with a short lived identity (machine id) generated by
tbot
via OIDC (in Github Actions). The provider seems to handle changes to Kubeconfig fine as long as the cluster info does not change however there is a condition where the provider is trying actions against existing resources before the Kubeconfig input has settled (via an output that yields a different value if you in CI or not) and this can cause it to completely fail when you switch between local and CI. Changing the path is the prime case where this happens because the provider saves the local system path which does not exist on other systems. I have managed to get this to a point where it works (providing the contents instead) but we end up with errors like the following when switching between local and CI:
Copy code
# on CI after local up:
If browser window does not open automatically, open it by clicking on the link:
     <http://127.0.0.1:38621/0f9740eb-81b8-4bd3-80e3-010159fec870>

# on local after CI up:
ERROR: no such file or directory
ERROR: no such file or directory
all because the provider does not wait for its inputs. Is this a bug?
I will raise an issue if this may be a bug
Actually I just found that the CI runs get the
Error: no such file or directory
error when run again because the location of the Machine Id identity file (an argument in the credential exec command) moves to a new temp location on each run.
I know some will say what is the problem if it works? Well... other users who may not be as deep in the weeds as I am will run into an issue and will report it as the issue when it is really a red herring that obscures the real issue.