https://pulumi.com logo
Join Slack
Powered by
There was a global DigitalOcean authentication inc...
# kubernetes
g
There was a global DigitalOcean authentication incident with their hosted Kubernetes. It was resolved quickly but I dont think I ran Pulumi since then. I'm now getting 
the server has asked for the client to provide credentials
I didnt have to use a new kubeconfig after they resolved the incident. Any ideas of what I can do?
When I do 
KUBECONFIG=~/Downloads/do-cluster-7b17531-kubeconfig.yaml pulumi preview
with the latest kubeconfig i get the same error.
s
Could it be possibly related to this? https://github.com/pulumi/pulumi-digitalocean/issues/445
g
possibly.. looking
What I dont understand is why I am still able to auth with k9s and look at the cluster, but pulumi cant.
s
Hmmm, that does seem odd. Is your Pulumi program using the default Kubernetes provider, or have you declared an explicit provider?
g
Copy code
const doK8sProvider = new k8s.Provider("do-k8s-provider", {
  kubeconfig
}, {
  parent: cluster
})
I did declare an explicit provider although i cant remember why
ah because cluster.provider didnt resolve to anything
it didnt seem like i could use the default provider with digital ocean
we use cluster.provider for AWS EKS with no problem
s
I haven’t seen your code, but typically when you create a cluster and then deploy workloads to that cluster in one (or more) Pulumi programs, you’ll end up using an explicit provider. The way AWS handles authentication for EKS in Kubeconfig appears to be different from the way DO handles it for DOKS, which may be why 
cluster.provider
works for EKS but not for DOKS. In any case, in reading through the issue I linked above and some related/linked issues, it sounds like running 
pulumi refresh
will re-generate a DO access token that will last for another seven days. Can you try running 
pulumi refresh
to see if that changes anything for you?
g
I cant run pulumi refresh because pulumi cant auth to the cluster
i get cluster unreachable
I think my issue is that my explicit provider is getting its kubeconfig from the cluster
Copy code
export const kubeconfig = pulumi.secret(cluster.kubeConfigs[0].rawConfig)

const doK8sProvider = new k8s.Provider("do-k8s-provider", {
  kubeconfig
}, {
  parent: cluster
})
and it expired or something
i did this because i wasnt able to do cluster.provider for all of the other objects so i decided to make an explicit one. then that caused me to store this kubeconfing in the provider
i will try the workaround in the github issue to see if i can construct one that works
ah yes i output this kubefconfig and its using a different one then the one i defined
and looking at the DO console i have a bunch of tokens
s
I suspected this might be the case. The workaround (create your own Kubeconfig with a user-supplied token) should take care of things for you.
g
the only issue seems to be that 
cluster.name
returns something different
cluster.name returns do-cluster-##### and i need it to be do-region-do-cluster-####
so im going to construct it manually but im unsure why this is happening
s
I’m not certain why the name might be returning something different, but constructing the name manually for your Kubeconfig should be fine. WRT to Kubernetes, it doesn’t really care about the name put in the Kubeconfig anyway.
g
well im using the code from the snippet and it isnt liking something and the only thing that is different is the name between the DO supplied kubeconfig and the one this code constructs (other than the token obviously)
Copy code
kubernetes:apps/v1:Deployment (kube-system/metrics-server):
    error: configured Kubernetes cluster is unreachable: unable to load Kubernetes client configuration from kubeconfig file. Make sure you have:

     	 • set up the provider as per <https://www.pulumi.com/registry/packages/kubernetes/installation-configuration/>

     invalid configuration: [context was not found for specified context: do-cluster-####, cluster has no server defined]

  pulumi:pulumi:Stack (oxheadinfra_do-dev):
    error: update failed
Copy code
// Manufacture a DO kubeconfig that uses a given API token.
//
// Note: this is slightly "different" than the default DOKS kubeconfig created
// for the cluster admin, which uses a new token automatically created by DO.
// <https://github.com/pulumi/pulumi-digitalocean/issues/312#issuecomment-1143432863>
export function createTokenKubeconfig(
  cluster: digitalocean.KubernetesCluster,
  user: pulumi.Input<string>,
  apiToken: pulumi.Input<string>,
): pulumi.Output<any> {
  const clusterName = cluster.name
  return pulumi.interpolate`apiVersion: v1
clusters:
- cluster:
  certificate-authority-data: ${cluster.kubeConfigs[0].clusterCaCertificate}
  server: ${cluster.endpoint}
name: ${cluster.name}
contexts:
- context:
  cluster: ${cluster.name}
  user: ${cluster.name}-${user}
name: ${cluster.name}
current-context: ${cluster.name}
kind: Config
users:
- name: ${cluster.name}-${user}
user:
  token: ${apiToken}
`;
}

const doCfg = new pulumi.Config("digitalocean")

export const kubeconfig = createTokenKubeconfig(cluster,"admin",doCfg.requireSecret("token"))

const doK8sProvider = new k8s.Provider("do-k8s-provider", {
  kubeconfig
}, {
  parent: cluster
})
my code ^^^
nope nevermind i dont think thats the issue
i replaced the name with my actual cluster name and its still bombing out with the same error
im copying and pasting in values from a working kubeconfig into the fuction
s
It looks like you may have a formatting issue in the Kubeconfig you shared above; the 
name
field needs to be indented:
Copy code
clusters:
- cluster:
    certificate-authority-data: ...
    server: ....
    name: ....
g
thank you for your help by the way
the formatting is a bit of a nightmare - that is copied directly from the comment
if i indent then i get "mapping of values is not allowed"
s
Happy to help. Let me double-check that against a working Kubeconfig…give me a few minutes to spin up a cluster.
g
i will be going AFK in 10 mins so dont put too much effort in
s
It’s OK, it’s not hard to spin one up real quick. It may not finish before you go AFK, but I’ll share it here and you can respond whenever you have time (even if that’s next week).
g
its possible the code in the comment is not working and just needs a couple tweaks
i indented 
name
and i get the 
cluster has no server defined
if i copy over directly from a working kubeconfig i get the mapping values error
s
OK, here’s a working 
clusters
section:
Copy code
clusters:
- cluster:
    certificate-authority-data: ...
    server: ....
  name: name
The 
name
field needs to be part of the list (array) created by the 
- cluster
line so it’s indented 2 spaces. The 
certificate-authority-data
and 
server
lines are indented 4 spaces.
Try that.
You’ll use the same indenting approach with defining your contexts and your users, BTW.
g
same error unfortunately
s
Very strange.
g
even if i copy directly from a working kubeconfig then it really goes wrong
Copy code
clusters:
- cluster:
    certificate-authority-data: ${cluster.kubeConfigs[0].clusterCaCertificate}
    server: ${cluster.endpoint}
  name: ${cluster.name}
Copy code
nvalid configuration: [context was not found for specified context: do-cluster-#####, cluster has no server defined]
s
Yep, that should be correct. Can you share the entire thing again?
g
Copy code
export function createTokenKubeconfig(
  cluster: digitalocean.KubernetesCluster,
  user: pulumi.Input<string>,
  apiToken: pulumi.Input<string>,
): pulumi.Output<any> {
  const clusterName = cluster.name
  return pulumi.interpolate`apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${cluster.kubeConfigs[0].clusterCaCertificate}
    server: ${cluster.endpoint}
  name: ${cluster.name}
contexts:
- context:
  cluster: ${cluster.name}
  user: ${cluster.name}-${user}
name: ${cluster.name}
current-context: ${cluster.name}
kind: Config
users:
- name: ${cluster.name}-${user}
user:
  token: ${apiToken}
`;
}
that?
s
OK. In the 
contexts
section, you need to indent 
cluster
and 
user
under 
- context:
, and 
name
should be part of the list of contexts (indented 2 spaces).
g
yes i had that also i just undid it because i thought we were going step by step sorry
s
The 
contexts
section should look (indentation wise) like the 
clusters
section.
g
oh wow now im getting 
already exists
errors
maybe i just need a refresh now
no nevermind its unreachable
well thats good i have already exists errors that means its authenticating
i must have not had the indentation right thanks
s
YAML indentation can be challenging at times
g
well especially when its in ticks
😂
any easy way to get around these 
already exists
errors besides importing everything?
s
Sorry, what sort of 
already exists
errors are you getting? Is this from a 
pulumi
command?
g
yes pulumi up
its like its trying to redeploy everything new now
s
Try 
pulumi refresh
first
g
yeah then i get unreachable error
and i dont want to do the delete envv ar
seems as is refresh is not using the new auth
ok how about i feed the new kubeconfig in to the pulumi preview
that seemed to work?
well it didnt error out, but it didnt give me an option to accept the replacements
sorry scratch that i was doing 
preview
not 
refresh
still same unreachable error
s
OK. The long-term fix may be to create a provider alias, since I think manually generating the Kubeconfig means Pulumi thinks it’s a new provider, and that’s why 
pulumi up
was trying to replace things. If feeding the new Kubeconfig causes 
pulumi preview
to work, then feeding it to 
pulumi up
should give you the option to accept any replacements (or not).
g
thanks for your help i have to run
s
Sure. Hit me up here in the community next week and let’s get you settled.
g
hey scott let me know if you have some time today to help me set up the provider alias
s
Hi Jimmy! I’m in meetings most of the day. I was thinking though…if you can feed the new Kubeconfig to 
pulumi preview
and it works (suggests replacements but doesn’t give you the option to accept them), then feeding the new Kubeconfig to 
pulumi up
should also work, and would give you the option to accept replacements. Ideally there will be only one replacement, which is the provider, and then we don’t have to worry about an alias. While I’m in meetings today, I’d suggest trying this to see what happens. (Don’t worry, you can always decline the operation if you feel it will be disruptive to your environment.) You could also use 
pulumi preview --diff
with the new Kubeconfig to get a more detailed indication of exactly what it would change; this might also give you a good indication of whether or not only the provider is changing. Post your findings back here, and I’ll check back in as I have time today between meetings.
g
SO preview works with and without the kubeconfig
I had incorrectly stated that REFRESH works with the kubeconfig but that is not the case
by feeding in the kubeconfig to an UP i get the duplicate error
it seems like it will not replace anything
I know you are b usy is there any chance you can escalate our ticket 3227? Ive had it open since 7/26 and havent heard back from the support person.
s
Oh, you have a ticket open? Yeah, I can definitely ask about an escalation.
g
yeah we pay for premium support
lol
i need some hand-holding via zoom. we're having another issue with pulumi not authing to an EKS cluster as well we're kinda dead here
s
I’ve asked internally for someone to look at your ticket. Heading into my meetings, but I’ll keep checking here and the internal thread and try to get some movement going.
g
you're good man no worries
i understand thanks for your time
i understand this is just the free support area haha
It looks like it wants to change provider and metadata? When I feed a kubeconfig into preview
I guess we arent paying for premium support so I have to be patient
s
Yeah, changing provider and metadata is what I would expect, but not changing any of the workloads deployed or the cluster itself.
If the preview only shows changes to provider and metadata, then you should be safe running an update operation.
g
What don you mean update? I cant run any ups.
s
Ah yes, that’s right---the 
up
reports duplicate errors, and refresh just plain doesn’t work.
g
Yep ups result in duplicate errors and refresh results in unreachable cluster
Phil in the ticket is suggesting we have to update the cluster provider with the new kubeconfig. The only issue is I dont think the cluster has a provider? At least cluster.provider for a DO kubernetes cluster doesnt resolve to anything.
Thats why I did a separate provider for installing the metrics server components.
s
I’m sure Phil knows more than I do, so let me see if I can connect with him to get a better understanding of what’s recommending (and why).
g
All good Scott whatever you think. You have been quite helpful.
s
Phil and I just chatted; I think he’s going to reach out to you shortly.
g
Awesome
any ideas? i sent the errors
s
Sorry, I thought Phil had already responded to you. Since you’re explicitly naming resources (instead of allowing Pulumi to auto-name), you’ll need to set 
deleteBeforeReplace
so 
pulumi up
can go ahead and update your Kubernetes resources.
g
Is that an arg i can feed in or something i have to put on every resource definition?
s
It’s a resource option that’s added to each resource definition: https://www.pulumi.com/docs/concepts/options/deletebeforereplace/
g
Awesome i’ll try it out. So the idea is it will see they exist and delete them and recreate them?
s
Yes, that’s the idea. The “duplicate exists” is coming because you’re supplying explicit names, and Pulumi can’t create the replacement first because a resource with that name already exists. Setting 
deleteBeforeReplace
tells the provider to do exactly that---delete the resource first, then create its replacement.
g
That sounds scary
Ill have to experiment
I have spaces buckets i cant lose the contents of
s
Yeah, this will delete resources and recreate them, so perform an appropriate level of testing and verification first.
g
Yeah ill have to be surgical
Thanks for the tip
Doesnt that only work if its in the state? It seems like the existing resources are not in the state hence the already exists error
Its a different error than when its a duplicate in the state
These errors are because the kubernetes api is telling pulumi they already exist
My understanding is deleteBeforeReplace is only for in-state pulumi duplication
s
You could dump your state to a JSON file (using 
pulumi state export
) and verify whether those resources are there or not. I haven’t seen the error messages and other outputs you’ve shared with Phil, so I’m only relaying what he passed on to me. I don’t know if your error messages are coming from Pulumi or from the Kubernetes API.
g
Im pretty sure its kubernetes saying it exists
I get the pulumi dupe errors all the time when i loop something and dont have a name increment or something like that
Is my understanding correct?
My fear is updating the kubeconfig changed all the resources outside the state
Couple of problems - k8s.helm.v3.chart and k8s.yaml.ConfigFile dont appear to have deleteBeforeReplace
i just ended up destroying the cluster and starting it back up with the provider with the static DO token kubeconfig
2 Views
Open in Slack
PreviousNext
Powered by