https://pulumi.com logo
Join Slack
Powered by
Hey, asking again. Anyone knows why `kubeconfig` ...
# kubernetes
m
Hey, asking again. Anyone knows why 
kubeconfig
is an option for the 
kubernetes
provider? It makes no sense. The string can be the kubeconfig contents or the kubeconfig local path. If it’s contents - it may contain login secrets, so we shouldn’t save it as a resource option If it’s a local path - then it will not be found on 
pulumi refresh
on a different machine Today 
pulumi refresh
is completely broken with 
kubernetes
b
it may contain login secrets, so we shouldn’t save it as a resource option
You can mark them as a secret
Copy code
const provider = k8s.Provider("example" {
  kubeconfig: pulumi.secret(kubeconfig)
})
If it’s a local path - then it will not be found on pulumi refresh on a different machine
yes, it’s not a good idea to make it a local path
m
But that’s the default now. When you do 
pulumi up
it overrides the local file path on the invoker’s machine When you do 
pulumi refresh
it reads the value from the config 
/home/NotMyUser/.kube/config
and tries to find it on the invokers machine - fails miserably with a very non indicative message:
Copy code
warning: configured Kubernetes cluster is unreachable: failed to parse kubeconfig data in `kubernetes:config:kubeconfig`- couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
    error: Preview failed: failed to read resource state due to unreachable cluster. If the cluster has been deleted, you can edit the pulumi state to remove this resource
If it fails finding the path locally - it tries to parse it as contents - I don’t understand how is this an acceptable fallback approach. Just too many wrongs here.
Copy code
// Note: the Python SDK was setting the kubeconfig value to "" by default, so explicitly check for empty string.
	if pathOrContents, ok := vars["kubernetes:config:kubeconfig"]; ok && pathOrContents != "" {
		var contents string

		// Handle the '~' character if it is set in the config string. Normally, this would be expanded by the shell
		// into the user's home directory, but we have to do that manually if it is set in a config value.
		if pathOrContents == "~" {
			// In case of "~", which won't be caught by the "else if"
			pathOrContents = homeDir()
		} else if strings.HasPrefix(pathOrContents, "~/") {
			pathOrContents = filepath.Join(homeDir(), pathOrContents[2:])
		}

		// If the variable is a valid filepath, load the file and parse the contents as a k8s config.
		_, err := os.Stat(pathOrContents)
		if err == nil {
			b, err := ioutil.ReadFile(pathOrContents)
			if err != nil {
				unreachableCluster(err)
			} else {
				contents = string(b)
			}
		} else { // Assume the contents are a k8s config.
			contents = pathOrContents
		}

		// Load the contents of the k8s config.
		apiConfig, err = clientcmd.Load([]byte(contents))
		if err != nil {
			unreachableCluster(err)
		} else {
			kubeconfig = clientcmd.NewDefaultClientConfig(*apiConfig, overrides)
			configurationNamespace, _, err := kubeconfig.Namespace()
			if err == nil {
				k.defaultNamespace = configurationNamespace
			}
		}
	}
I’m trying to understand why 
kubeconfig
is even necessary if it is always overridden on 
pulumi up
. What is the reason we save it as a provider resource option
b
so you’re saving the path in the provider is not honoured when you refresh? if so, that’s a bug
and you should file an issue with a repro
m
Worse - it is honored in refresh - that’s why it fails
It’s not honored in 
pulumi up
that’s why 
pulumi up
always works, it uses the the local kubeconfig and overrides the path of the resource too
q
Sorry for interrupting the discussion but @billowy-army-68599 will you please share the pulumi-python equivalent to the one below where the kubeconfig can be marked a secret?
Copy code
const provider = k8s.Provider("example" {
  kubeconfig: pulumi.secret(kubeconfig)
})
b
something like this
Copy code
import pulumi
import pulumi_kubernetes as k8s

kubeconfig = pulumi.Output.secret(f = open('kubeconfig.json'))

provider = k8s.provider.Provider(
    kubeconfig=kubeconfig
)
(untested)
q
I'll test it out...thank you! 🙂
m
@billowy-army-68599 I can open an issue in 
pulumi-kubernetes
but I wanted to make sure this is indeed a bug and not something I don’t understand. 
Kubeconfig
gets populated by default with a path that breaks 
pulumi refresh
on different machines. In addition every 
pulumi up
ignores 
Kubeconfig
and overwrites it. So for my perspective - 
Kubeconfig
should be deleted from the provider, as it makes no sense, but there is so much code around that in 
pulumi-kubernetes
that makes me think that I might be missing something. Either no one uses 
pulumi-kubernetes
Either no one uses 
pulumi refresh
with 
pulumi-kubernetes
Either everyone that runs 
pulumi-kubernetes
and uses 
pulumi refresh
does it from the same machine. Just wanted so ask around the community before I open a PR
b
Kubeconfig gets populated by default with a path that breaks pulumi refresh on different machines.
This is expected behaviour the path to a kubeconfig gets stored in your state.
Either no one uses pulumi-kubernetes
obviously not true 🙂
Either no one uses pulumi refresh with pulumi-kubernetes
Also not true 🙂
Either everyone that runs pulumi-kubernetes and uses pulumi refresh does it from the same machine.
Not likely The actual reason is more like
Everyone who uses pulumi-kubernetes doesn’t hardcode paths in the kubeconfig. They either use the ambient provider (ie, using the 
KUBECONFIG
environment variable) or they pass the kubeconfig from a created cluster
m
But what is the meaning of storing a path to Kubeconfig (which is the default behavior) if its local to a specific machine. We don’t hardcode the path, we use 
KUBECONFIG
environment variable, and then the path gets resolved into the provider state. A refresh on a different machine uses that provider’s state, but will not find the kubeconfig file - which, by the way, is not guaranteed to have the same information. So saving a path (default) in Kubeconfig doesn’t make much sense. Saving the contents of the local Kubeconfig file in that state makes more sense, since some of the cluster information is saved there, but then on the other hand, part of the Kubeconfig has secret auth tokens (with expiry) that probably shouldn’t be saved by Pulumi. --- So let’s say we want to fix 
pulumi refresh
failing for us. We have two options: One - without code changes: 1. Use same path on all machines that will store the configuration for our cluster - a bit non standard 2. Use unexpanded kubeconfig=“~/.kube/config” - slightly better option that 1 since it will default to different home dir across systems 3. Put the contents of the Kubeconfig in the configuration - I don’t think it will work nicely after the credentials expire. So option number 2 makes most sense, but we need to make sure across all developer/CI machines that we use ~/.kube/config as the path Two - with code changes: 1. Read local kubeconfig on 
pulumi refresh
- not sure how to prioritize between local one to saved one. Regardless - we need to fix: 1. Non indicative error message that happens when saved kubeconfig path is not found locally / or access to it is denied 2. Fix the “if file was not found, treat file path as Kubeconfig contents and fail miserably. I would first try to understand if its contents by checking if it starts with “apiVersion:” or something (maybe if it contains \n). Then if its a single line file path - then treat it as a file and if its not found - write that it’s not found. Need to be careful around printing secrets though.
Actually it makes sense that if people do: 
KUBECONFIG=~/.kube/config
This bug might never arise for refresh. But if people: 
KUBECONFIG=/home/sam/.kube/config
Then 
pulumi refresh
will not work on other machines (without the user sam and access to sam’s folder)
b
are you saying you’re not defining an explicit provider and it’s still storing the kubeconfig path?
if so, that’s absolutely a bug
m
Depends what you mean by explicit: 
KUBECONFIG=/path pulumi up
Will go through each SDK differently But they all do the same:
Copy code
if isZero(args.Kubeconfig) {
		args.Kubeconfig = pulumi.StringPtr(getEnvOrDefault("", nil, "KUBECONFIG").(string))
	}
Copy code
KUBECONFIG=~/.kube/config pulumi up
Will be saved as 
Kubeconfig: ~/.kube/config
and then 
pulumi refresh
will work on other machines that use home dir
Copy code
KUBECONFIG=/home/sam/.kube/config pulumi up
Which might be expanded in the shell by mistake Will be saved as 
Kubeconfig: /home/sam/.kube/config
and then 
pulumi refresh
won’t work on other machines that do not belong to sam
Copy code
KUBECONFIG= pulumi up
Will be saved as 
Kubeconfig: ""
and looks like 
pulumi refresh
should work. Since
Copy code
if pathOrContents, ok := vars["kubernetes:config:kubeconfig"]; ok && pathOrContents != "" {
is false. but I think it didn’t work on 
pulumi up
since kubeconfig was not found
Maybe 99.9% of 
pulumi-kubernetes
users have:
Copy code
KUBECONFIG=~/.kube/config
configured. And we stepped on the:
Copy code
KUBECONFIG=/home/sam/.kube/config
landmine In any case the error message needs to be fixed from:
Copy code
warning: configured Kubernetes cluster is unreachable: failed to parse kubeconfig data in `kubernetes:config:kubeconfig`- couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
    error: Preview failed: failed to read resource state due to unreachable cluster. If the cluster has been deleted, you can edit the pulumi state to remove this resource
785 Views
Open in Slack
PreviousNext
Powered by