Ok, I have a few big blockers unfortunately and co...
# kubernetes
k
Ok, I have a few big blockers unfortunately and could really use some guidance on these. I’ve mentioned some of these previously but I’ll list them here with the context that I’ve gathered while digging… 1. GCP (GKE) Kube cluster is currently recreated every time I run pulumi up a. I am attempting to understand the diff in order to remove anything that changes and causes it to recreate b. On the second and subsequent runs, the oauth scopes appear to be deleted - could this cause it? or is this normal and just sending me the wrong direction?
Copy code
~ nodeConfig: {
          ~ oauthScopes: [
              - [0]: "<https://www.googleapis.com/auth/monitoring>"
              - [1]: "<https://www.googleapis.com/auth/logging.write>"
            ]
        }
2. I’m trying to generate a kubeconfig and use that to create a provider for my gke cluster however every time I get an error for line 5 in the yaml.
Copy code
error: rpc error: code = Unknown desc = failed to parse kubeconfig: yaml: line 5: found character that cannot start any token
Code is like this:
Copy code
func generateKubeconfig(clusterEndpoint pulumi.StringOutput, clusterName pulumi.StringOutput,
	clusterMasterAuth container.ClusterMasterAuthOutput) pulumi.StringOutput {
	context := pulumi.Sprintf("dimo_%s", clusterName)
	clusterCaCertificate := clusterMasterAuth.ClusterCaCertificate().Elem()
	fmt.Printf("(gen config) clusterCaCertificate: %v", clusterCaCertificate)
	fmt.Printf("(gen config) clusterEndpoint: %v", clusterEndpoint)

	return pulumi.Sprintf(`apiVersion: v1
clusters:
- cluster:
		certificate-authority-data: %v
		server: <https://%v>
	name: %s
contexts:
- context:
    cluster: %s
    user: %s
  name: %s
current-context: %s
kind: Config
preferences: {}
users:
- name: %s
  user:
    exec:
      apiVersion: <http://client.authentication.k8s.io/v1beta1|client.authentication.k8s.io/v1beta1>
      command: gke-gcloud-auth-plugin
      installHint: Install gke-gcloud-auth-plugin for use with kubectl by following
        <https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke>
      provideClusterInfo: true
`, clusterCaCertificate, clusterEndpoint, context, context, context, context, context, context)
}
You can see the full context here: https://github.com/phutchins/dimo-node/blob/main/infrastructure/k8s_provider_gke.go
And regarding the kubeconfig and yaml error, I’ve made sure that there are no tabs in my code so I don’t believe its that.
If I could print the kubeconfig, I think I could troubleshoot but I can’t find a way to do that. I always end up with {0x0flj2ij3flk23f] type response which I assume is either a memory address or object that is not fully returned or something.
You can see that here:
KubeConfig: {0x1400004a380}(pall) Args[0]: dimo-dev-401815-e2c3d0d(pall) Args[1]: 35.192.120.34
I’ve added the oauth scopes to the config and that seems to have stopped the cluster recreation (maybe).
Looks like something is putting tabs in my kubeconfig. I’ve confirmed only spaces in the file even after save…
Copy code
"plaintext": "\"\\napiVersion: v1\\nclusters:\\n- cluster:\\n\\tcertificate-authority-data:
However, it’s complaining about line 5 which is the server line and is after cluster and certificate-authority-data lines.
r
Hi. Your
name
field does not belong to server section.
Is it server name?
k
It belongs to the cluster section. That should be in the right place. I compare it to my actual kube/config and it looks right (in the code at least, and even in the logs from what I can tell).
I’ve gotten everything else working except for that line 5 error. Using
pulumi stack export --show-secrets
has been really helpful to see the output of the compiled kubeconfig. Just can’t figure out how to get rid of the tabs! Or if thats even the actual problem… Is there anything in Pulumi that adds those tab characters? Or anything in Pulumi that removes them before using the YAML with kubectl?
If I simply copy the kubeconfig from the logs, it works just fine (of course I have no tabs in there and only spaces. Whats odd tho is that if I move lines, i.e. swap line 4 and 5, its still always line 5 that errors which seems fishy to me.
@salmon-account-74572 you happen to have an idea who might be able to point me in the right direction with this one? I’m stumped at this point digging through the library code…
Ok, finally got this resolved. It was due to a \t character that made its way into my go file because of my IDE. Problem is that pulumi cached it and didn’t replace the resource on changes even after a pulumi destroy which was incredibly misleading.
s
@kind-fireman-33438 That’s indeed very odd! So the random
\t
character was in the Kubeconfig, which in turn was used to create an explicit provider, is that right?