https://pulumi.com logo
Join Slack
Powered by
What is the recommended way to deploy Kubernetes m...
# kubernetes
n
What is the recommended way to deploy Kubernetes manifests from YAML files now? For some context on what I tried: Previously the 
kubernetes.yaml
(ie 
@pulumi/kubernetes
) module wasn’t working because things weren’t propagating 
opts.provider
. So I made a workaround that uses 
local.Command
and setting 
args.environment.KUBECONFIG
, but now that stopped working too for some reason. It seems to ignore the environment variable (and it tries to use the default kubeconfig file, but luckily I’ve added some guardrails to avoid disasters in case such a bug pops up). I remember seeing that the propagation bug was fixed. So, I tried to convert to use the builtin yaml methods again, but it looks like nothing in the 
kubernetes.yaml
module works now, because it complains that 
map
can’t be called on undefined in yaml/yaml.ts:2993. These are multi-document yaml files (cert manager and aws lb controller), if it matters?
q
Can you share the code you used that failed for kubernetes.yaml.ConfigFile, please.
👍 1
n
I’ll try and make a small repro repo and link it here
Heya, sorry got distracted. I stripped most of it out https://github.com/omninonsense/pulumi-repro
I didn’t wanna spend a lot of time untangling the RBAC/SSO stuff, so i left that in, but just ignore those parts
Thanks for looking into this btw
q
It'll be Monday now, as it's quite late in the UK. But I'll give this a go and get back to you
n
Oh yeah, no worries, I assumed it would be next week anyway 🙂
(from the UK as well, btw haha)
🎉 1
Hope you have a fun weekend! 😁
q
You too. Cheers
OK. So the cert manager YAML is fine and I can deploy it no problem
The problem is your provider
Copy code
new k8s.yaml.ConfigFile(
      "cert-manager",
      {
        file: "k8s/cert-manager.yaml",
      },
      { provider: this.cluster.provider }
    );
You can’t actually use 
this.cluster.provider
Instead, you should create a Kubernetes provider like so:
Copy code
const provider = new k8s.Provider("eks", { kubeconfig: this.cluster.kubeconfig });
and then use that provider with any Kubernetes resource you want to deploy
n
Oooh… That was silly of me in all fairness, because IIRC at some point I started writing something that used 
k8s.Provider
but then I saw 
cluster.provider
and thought “oh how convenient, I’ll use that” 🤦‍♂️
q
Yeah, the 
cluster.provider
is the AWS provider used to spin up the cluster 🙂
Easy mistake to make!
n
Ah!
Okay that makes a lot of sense
Thank you for looking into this!
Oh hmm, it seems the comment on the 
eks.Cluster.provider
class led me to attempt to use it. It’s also typed as 
k8s.Provider
, which is probably why TypeScript didn’t complain
Copy code
/**
     * A Kubernetes resource provider that can be used to deploy into this cluster. For example, the code below will
     * create a new Pod in the EKS cluster.
     *
     *     let eks = new Cluster("eks");
     *     let pod = new kubernetes.core.v1.Pod("pod", { ... }, { provider: eks.provider });
     *
     */
So I’ve refactored it to this approach, but I’m getting the same error. Also, I forgot that a few “manually” created resources (eg there’s a few 
k8s.rbac.v1.RoleBinding
that use it in repro repo) worked when using 
this.cluster.provider
I wonder if the yaml parser doesn’t support CRDs (since 
kube2pulumi
doesn’t either; it gave me a hint to use the 
crd2pulumi
tool instead)
Let me try adding a simpler resource
Okay, it seems that even a single deployment caused the same error
I’ve just used this example deployment and put it in `example.yaml`: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment
@quiet-wolf-18467 would it help if I opened an issue on Github, if this is something that might take longer to investigate?
Or alternatively, if you have an internal system, feel free to use/clone the repro repo as an example
Also, this bit was wrong; it was a bug on my end:
So I made a workaround that uses 
local.Command
and setting 
args.environment.KUBECONFIG
, but now that stopped working too for some reason. It seems to ignore the environment variable (and it tries to use the default kubeconfig file, but luckily I’ve added some guardrails to avoid disasters in case such a bug pops up).
😂 I was refactoring/renaming a few things, and there was a typo that slipped through because of an 
any
q
Hmm, I had no idea 
.provider
was supposed to be a Kubernetes provider
Can you open an issue on the GitHub repo and we’ll take a look?
👍 1
n
Yup, no problem
Also, this is from the awsx/eks packages (not the aws one). I don’t know if there’s an EKS equivalent in the AWS one, so maybe that’s where the confusion happened
Uuh would it fit better into pulumi/pulumi, or pulumi/pulumi-kubernetes?
Id say kubernetes, but wanted to double check
Meh, we can always move it 🤷 I’ll open in the kubernetes one for now
8 Views
Open in Slack
PreviousNext
Powered by