https://pulumi.com logo
Powered by
g

glamorous-australia-21342

07/26/2023, 6:18 PM
There was a global DigitalOcean authentication incident with their hosted Kubernetes. It was resolved quickly but I dont think I ran Pulumi since then. I'm now getting 
the server has asked for the client to provide credentials
I didnt have to use a new kubeconfig after they resolved the incident. Any ideas of what I can do?
When I do 
KUBECONFIG=~/Downloads/do-cluster-7b17531-kubeconfig.yaml pulumi preview
with the latest kubeconfig i get the same error.
s

salmon-account-74572

07/26/2023, 6:55 PM
Could it be possibly related to this? https://github.com/pulumi/pulumi-digitalocean/issues/445
g

glamorous-australia-21342

07/26/2023, 7:06 PM
possibly.. looking
What I dont understand is why I am still able to auth with k9s and look at the cluster, but pulumi cant.
s

salmon-account-74572

07/27/2023, 7:36 PM
Hmmm, that does seem odd. Is your Pulumi program using the default Kubernetes provider, or have you declared an explicit provider?
g

glamorous-australia-21342

07/27/2023, 7:37 PM
Copy code
const doK8sProvider = new k8s.Provider("do-k8s-provider", {
  kubeconfig
}, {
  parent: cluster
})
I did declare an explicit provider although i cant remember why
ah because cluster.provider didnt resolve to anything
it didnt seem like i could use the default provider with digital ocean
we use cluster.provider for AWS EKS with no problem
s

salmon-account-74572

07/27/2023, 7:45 PM
I haven’t seen your code, but typically when you create a cluster and then deploy workloads to that cluster in one (or more) Pulumi programs, you’ll end up using an explicit provider. The way AWS handles authentication for EKS in Kubeconfig appears to be different from the way DO handles it for DOKS, which may be why 
cluster.provider
works for EKS but not for DOKS. In any case, in reading through the issue I linked above and some related/linked issues, it sounds like running 
pulumi refresh
will re-generate a DO access token that will last for another seven days. Can you try running 
pulumi refresh
to see if that changes anything for you?
g

glamorous-australia-21342

07/27/2023, 8:24 PM
I cant run pulumi refresh because pulumi cant auth to the cluster
i get cluster unreachable
I think my issue is that my explicit provider is getting its kubeconfig from the cluster
Copy code
export const kubeconfig = pulumi.secret(cluster.kubeConfigs[0].rawConfig)

const doK8sProvider = new k8s.Provider("do-k8s-provider", {
  kubeconfig
}, {
  parent: cluster
})
and it expired or something
i did this because i wasnt able to do cluster.provider for all of the other objects so i decided to make an explicit one. then that caused me to store this kubeconfing in the provider
i will try the workaround in the github issue to see if i can construct one that works
ah yes i output this kubefconfig and its using a different one then the one i defined
and looking at the DO console i have a bunch of tokens
s

salmon-account-74572

07/28/2023, 7:12 PM
I suspected this might be the case. The workaround (create your own Kubeconfig with a user-supplied token) should take care of things for you.
g

glamorous-australia-21342

07/28/2023, 7:12 PM
the only issue seems to be that 
cluster.name
returns something different
cluster.name returns do-cluster-##### and i need it to be do-region-do-cluster-####
so im going to construct it manually but im unsure why this is happening
s

salmon-account-74572

07/28/2023, 7:14 PM
I’m not certain why the name might be returning something different, but constructing the name manually for your Kubeconfig should be fine. WRT to Kubernetes, it doesn’t really care about the name put in the Kubeconfig anyway.
g

glamorous-australia-21342

07/28/2023, 7:14 PM
well im using the code from the snippet and it isnt liking something and the only thing that is different is the name between the DO supplied kubeconfig and the one this code constructs (other than the token obviously)
Copy code
kubernetes:apps/v1:Deployment (kube-system/metrics-server):
    error: configured Kubernetes cluster is unreachable: unable to load Kubernetes client configuration from kubeconfig file. Make sure you have:

     	 • set up the provider as per <https://www.pulumi.com/registry/packages/kubernetes/installation-configuration/>

     invalid configuration: [context was not found for specified context: do-cluster-####, cluster has no server defined]

  pulumi:pulumi:Stack (oxheadinfra_do-dev):
    error: update failed
Copy code
// Manufacture a DO kubeconfig that uses a given API token.
//
// Note: this is slightly "different" than the default DOKS kubeconfig created
// for the cluster admin, which uses a new token automatically created by DO.
// <https://github.com/pulumi/pulumi-digitalocean/issues/312#issuecomment-1143432863>
export function createTokenKubeconfig(
  cluster: digitalocean.KubernetesCluster,
  user: pulumi.Input<string>,
  apiToken: pulumi.Input<string>,
): pulumi.Output<any> {
  const clusterName = cluster.name
  return pulumi.interpolate`apiVersion: v1
clusters:
- cluster:
  certificate-authority-data: ${cluster.kubeConfigs[0].clusterCaCertificate}
  server: ${cluster.endpoint}
name: ${cluster.name}
contexts:
- context:
  cluster: ${cluster.name}
  user: ${cluster.name}-${user}
name: ${cluster.name}
current-context: ${cluster.name}
kind: Config
users:
- name: ${cluster.name}-${user}
user:
  token: ${apiToken}
`;
}

const doCfg = new pulumi.Config("digitalocean")

export const kubeconfig = createTokenKubeconfig(cluster,"admin",doCfg.requireSecret("token"))

const doK8sProvider = new k8s.Provider("do-k8s-provider", {
  kubeconfig
}, {
  parent: cluster
})
my code ^^^
nope nevermind i dont think thats the issue
i replaced the name with my actual cluster name and its still bombing out with the same error
im copying and pasting in values from a working kubeconfig into the fuction
s

salmon-account-74572

07/28/2023, 7:26 PM
It looks like you may have a formatting issue in the Kubeconfig you shared above; the 
name
field needs to be indented:
Copy code
clusters:
- cluster:
    certificate-authority-data: ...
    server: ....
    name: ....
g

glamorous-australia-21342

07/28/2023, 7:26 PM
thank you for your help by the way
the formatting is a bit of a nightmare - that is copied directly from the comment
if i indent then i get "mapping of values is not allowed"
s

salmon-account-74572

07/28/2023, 7:27 PM
Happy to help. Let me double-check that against a working Kubeconfig…give me a few minutes to spin up a cluster.
g

glamorous-australia-21342

07/28/2023, 7:27 PM
i will be going AFK in 10 mins so dont put too much effort in
s

salmon-account-74572

07/28/2023, 7:28 PM
It’s OK, it’s not hard to spin one up real quick. It may not finish before you go AFK, but I’ll share it here and you can respond whenever you have time (even if that’s next week).
g

glamorous-australia-21342

07/28/2023, 7:28 PM
its possible the code in the comment is not working and just needs a couple tweaks
i indented 
name
and i get the 
cluster has no server defined
if i copy over directly from a working kubeconfig i get the mapping values error
s

salmon-account-74572

07/28/2023, 7:31 PM
OK, here’s a working 
clusters
section:
Copy code
clusters:
- cluster:
    certificate-authority-data: ...
    server: ....
  name: name
The 
name
field needs to be part of the list (array) created by the 
- cluster
line so it’s indented 2 spaces. The 
certificate-authority-data
and 
server
lines are indented 4 spaces.
Try that.
You’ll use the same indenting approach with defining your contexts and your users, BTW.
g

glamorous-australia-21342

07/28/2023, 7:32 PM
same error unfortunately
s

salmon-account-74572

07/28/2023, 7:32 PM
Very strange.
g

glamorous-australia-21342

07/28/2023, 7:32 PM
even if i copy directly from a working kubeconfig then it really goes wrong
Copy code
clusters:
- cluster:
    certificate-authority-data: ${cluster.kubeConfigs[0].clusterCaCertificate}
    server: ${cluster.endpoint}
  name: ${cluster.name}
Copy code
nvalid configuration: [context was not found for specified context: do-cluster-#####, cluster has no server defined]
s

salmon-account-74572

07/28/2023, 7:33 PM
Yep, that should be correct. Can you share the entire thing again?
g

glamorous-australia-21342

07/28/2023, 7:33 PM
Copy code
export function createTokenKubeconfig(
  cluster: digitalocean.KubernetesCluster,
  user: pulumi.Input<string>,
  apiToken: pulumi.Input<string>,
): pulumi.Output<any> {
  const clusterName = cluster.name
  return pulumi.interpolate`apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${cluster.kubeConfigs[0].clusterCaCertificate}
    server: ${cluster.endpoint}
  name: ${cluster.name}
contexts:
- context:
  cluster: ${cluster.name}
  user: ${cluster.name}-${user}
name: ${cluster.name}
current-context: ${cluster.name}
kind: Config
users:
- name: ${cluster.name}-${user}
user:
  token: ${apiToken}
`;
}
that?
s

salmon-account-74572

07/28/2023, 7:35 PM
OK. In the 
contexts
section, you need to indent 
cluster
and 
user
under 
- context:
, and 
name
should be part of the list of contexts (indented 2 spaces).
g

glamorous-australia-21342

07/28/2023, 7:35 PM
yes i had that also i just undid it because i thought we were going step by step sorry
s

salmon-account-74572

07/28/2023, 7:35 PM
The 
contexts
section should look (indentation wise) like the 
clusters
section.
g

glamorous-australia-21342

07/28/2023, 7:35 PM
oh wow now im getting 
already exists
errors
maybe i just need a refresh now
no nevermind its unreachable
well thats good i have already exists errors that means its authenticating
i must have not had the indentation right thanks
s

salmon-account-74572

07/28/2023, 7:36 PM
YAML indentation can be challenging at times
g

glamorous-australia-21342

07/28/2023, 7:38 PM
well especially when its in ticks
😂
any easy way to get around these 
already exists
errors besides importing everything?
s

salmon-account-74572

07/28/2023, 7:44 PM
Sorry, what sort of 
already exists
errors are you getting? Is this from a 
pulumi
command?
g

glamorous-australia-21342

07/28/2023, 7:44 PM
yes pulumi up
its like its trying to redeploy everything new now
s

salmon-account-74572

07/28/2023, 7:44 PM
Try 
pulumi refresh
first
g

glamorous-australia-21342

07/28/2023, 7:45 PM
yeah then i get unreachable error
and i dont want to do the delete envv ar
seems as is refresh is not using the new auth
ok how about i feed the new kubeconfig in to the pulumi preview
that seemed to work?
well it didnt error out, but it didnt give me an option to accept the replacements
sorry scratch that i was doing 
preview
not 
refresh
still same unreachable error
s

salmon-account-74572

07/28/2023, 7:48 PM
OK. The long-term fix may be to create a provider alias, since I think manually generating the Kubeconfig means Pulumi thinks it’s a new provider, and that’s why 
pulumi up
was trying to replace things. If feeding the new Kubeconfig causes 
pulumi preview
to work, then feeding it to 
pulumi up
should give you the option to accept any replacements (or not).
g

glamorous-australia-21342

07/28/2023, 7:48 PM
thanks for your help i have to run
s

salmon-account-74572

07/28/2023, 7:49 PM
Sure. Hit me up here in the community next week and let’s get you settled.
g

glamorous-australia-21342

08/01/2023, 3:13 PM
hey scott let me know if you have some time today to help me set up the provider alias
s

salmon-account-74572

08/01/2023, 3:31 PM
Hi Jimmy! I’m in meetings most of the day. I was thinking though…if you can feed the new Kubeconfig to 
pulumi preview
and it works (suggests replacements but doesn’t give you the option to accept them), then feeding the new Kubeconfig to 
pulumi up
should also work, and would give you the option to accept replacements. Ideally there will be only one replacement, which is the provider, and then we don’t have to worry about an alias. While I’m in meetings today, I’d suggest trying this to see what happens. (Don’t worry, you can always decline the operation if you feel it will be disruptive to your environment.) You could also use 
pulumi preview --diff
with the new Kubeconfig to get a more detailed indication of exactly what it would change; this might also give you a good indication of whether or not only the provider is changing. Post your findings back here, and I’ll check back in as I have time today between meetings.
g

glamorous-australia-21342

08/01/2023, 3:32 PM
SO preview works with and without the kubeconfig
I had incorrectly stated that REFRESH works with the kubeconfig but that is not the case
by feeding in the kubeconfig to an UP i get the duplicate error
it seems like it will not replace anything
I know you are b usy is there any chance you can escalate our ticket 3227? Ive had it open since 7/26 and havent heard back from the support person.
s

salmon-account-74572

08/01/2023, 3:49 PM
Oh, you have a ticket open? Yeah, I can definitely ask about an escalation.
g

glamorous-australia-21342

08/01/2023, 3:49 PM
yeah we pay for premium support
lol
i need some hand-holding via zoom. we're having another issue with pulumi not authing to an EKS cluster as well we're kinda dead here
s

salmon-account-74572

08/01/2023, 3:53 PM
I’ve asked internally for someone to look at your ticket. Heading into my meetings, but I’ll keep checking here and the internal thread and try to get some movement going.
g

glamorous-australia-21342

08/01/2023, 3:54 PM
you're good man no worries
i understand thanks for your time
i understand this is just the free support area haha
It looks like it wants to change provider and metadata? When I feed a kubeconfig into preview
I guess we arent paying for premium support so I have to be patient
s

salmon-account-74572

08/02/2023, 1:32 PM
Yeah, changing provider and metadata is what I would expect, but not changing any of the workloads deployed or the cluster itself.
If the preview only shows changes to provider and metadata, then you should be safe running an update operation.
g

glamorous-australia-21342

08/02/2023, 1:46 PM
What don you mean update? I cant run any ups.
s

salmon-account-74572

08/02/2023, 2:02 PM
Ah yes, that’s right---the 
up
reports duplicate errors, and refresh just plain doesn’t work.
g

glamorous-australia-21342

08/02/2023, 2:03 PM
Yep ups result in duplicate errors and refresh results in unreachable cluster
Phil in the ticket is suggesting we have to update the cluster provider with the new kubeconfig. The only issue is I dont think the cluster has a provider? At least cluster.provider for a DO kubernetes cluster doesnt resolve to anything.
Thats why I did a separate provider for installing the metrics server components.
s

salmon-account-74572

08/02/2023, 3:01 PM
I’m sure Phil knows more than I do, so let me see if I can connect with him to get a better understanding of what’s recommending (and why).
g

glamorous-australia-21342

08/02/2023, 3:02 PM
All good Scott whatever you think. You have been quite helpful.
s

salmon-account-74572

08/02/2023, 4:39 PM
Phil and I just chatted; I think he’s going to reach out to you shortly.
g

glamorous-australia-21342

08/02/2023, 4:58 PM
Awesome
any ideas? i sent the errors
s

salmon-account-74572

08/03/2023, 9:34 PM
Sorry, I thought Phil had already responded to you. Since you’re explicitly naming resources (instead of allowing Pulumi to auto-name), you’ll need to set 
deleteBeforeReplace
so 
pulumi up
can go ahead and update your Kubernetes resources.
g

glamorous-australia-21342

08/03/2023, 9:35 PM
Is that an arg i can feed in or something i have to put on every resource definition?
s

salmon-account-74572

08/03/2023, 9:38 PM
It’s a resource option that’s added to each resource definition: https://www.pulumi.com/docs/concepts/options/deletebeforereplace/
g

glamorous-australia-21342

08/03/2023, 9:45 PM
Awesome i’ll try it out. So the idea is it will see they exist and delete them and recreate them?
s

salmon-account-74572

08/03/2023, 9:47 PM
Yes, that’s the idea. The “duplicate exists” is coming because you’re supplying explicit names, and Pulumi can’t create the replacement first because a resource with that name already exists. Setting 
deleteBeforeReplace
tells the provider to do exactly that---delete the resource first, then create its replacement.
g

glamorous-australia-21342

08/03/2023, 9:48 PM
That sounds scary
Ill have to experiment
I have spaces buckets i cant lose the contents of
s

salmon-account-74572

08/03/2023, 9:49 PM
Yeah, this will delete resources and recreate them, so perform an appropriate level of testing and verification first.
g

glamorous-australia-21342

08/03/2023, 9:52 PM
Yeah ill have to be surgical
Thanks for the tip
Doesnt that only work if its in the state? It seems like the existing resources are not in the state hence the already exists error
Its a different error than when its a duplicate in the state
These errors are because the kubernetes api is telling pulumi they already exist
My understanding is deleteBeforeReplace is only for in-state pulumi duplication
s

salmon-account-74572

08/03/2023, 9:58 PM
You could dump your state to a JSON file (using 
pulumi state export
) and verify whether those resources are there or not. I haven’t seen the error messages and other outputs you’ve shared with Phil, so I’m only relaying what he passed on to me. I don’t know if your error messages are coming from Pulumi or from the Kubernetes API.
g

glamorous-australia-21342

08/03/2023, 9:58 PM
Im pretty sure its kubernetes saying it exists
I get the pulumi dupe errors all the time when i loop something and dont have a name increment or something like that
Is my understanding correct?
My fear is updating the kubeconfig changed all the resources outside the state
Couple of problems - k8s.helm.v3.chart and k8s.yaml.ConfigFile dont appear to have deleteBeforeReplace
i just ended up destroying the cluster and starting it back up with the provider with the static DO token kubeconfig
2 Views
Powered by