https://pulumi.com logo
Powered by
e

early-musician-41645

11/05/2018, 7:06 PM
We had a thread about how a helm-chart I'm installing fails because it's trying to install something that already exists, e.g. ServiceAccount named 
aws-node
. I thought that it was managed outside the pulumi stack and raising the error, but that's not the case. I've now confirmed that it's failing even if the entire stack is managed by Pulumi. I first destroyed all the resources, then I 
pulumi up
which creates an EKS cluster and a helm chart. The helm chart has a ServiceAccount and it fails with this:
Copy code
Diagnostics:
  kubernetes:core:ServiceAccount (kube-system/aws-node):
    error: Plan apply failed: serviceaccounts "aws-node" already exists
The expected behavior is that the existing ServiceAccount would not raise an error and instead would get treated as a no-op.
c

creamy-potato-29402

11/05/2018, 7:18 PM
@early-musician-41645 Sorry, I’m confused… you did not delete that SA, right? And so the chart is trying to bring it up, but it can’t because it already exists?
e

early-musician-41645

11/05/2018, 7:19 PM
The new EKS cluster creates the SA initially. After cluster creation I see Pulumi trying to create the helm chart resources. At that point 
pulumi up
throws an error.
c

creamy-potato-29402

11/05/2018, 7:20 PM
which helm chart? Ours or yours?
e

early-musician-41645

11/05/2018, 7:20 PM
I'm using 
new eks.Cluster
. One of the things created by that is the SA
I'm then trying to deploy my own Helm chart which also declares the same SA, but meant as an upgrade, install if it doesn't exist
c

creamy-potato-29402

11/05/2018, 7:21 PM
hmmmm
This is a use case I had not considered.
e

early-musician-41645

11/05/2018, 7:22 PM
I'm basically trying to manage EKS resources that are already created, but need to be tweaked.
c

creamy-potato-29402

11/05/2018, 7:22 PM
It might be worth having an idea of “overlays”
e

early-musician-41645

11/05/2018, 7:22 PM
e.g. the CNI plugin
c

creamy-potato-29402

11/05/2018, 7:22 PM
Right, right.
So the way to do this normally would be the transforms API.
e

early-musician-41645

11/05/2018, 7:22 PM
overlays, or support an "upgrade" flag on the helm provider
c

creamy-potato-29402

11/05/2018, 7:22 PM
@microscopic-florist-22719 did we ever implement that?
I don’t see it in the API.
m

microscopic-florist-22719

11/05/2018, 7:24 PM
did we ever implement which? injecting resources?
c

creamy-potato-29402

11/05/2018, 7:24 PM
transforms on the EKS library
@early-musician-41645 wants to mutate the EKS kube resources, basically.
m

microscopic-florist-22719

11/05/2018, 7:25 PM
we have not implemented any support for something like this, no. I'm working on decomposing the EKS library a bit s.t. things like this are easier to do
c

creamy-potato-29402

11/05/2018, 7:26 PM
@early-musician-41645 at a high level do you think it is a problem to change the kube resources when the EKS cluster is built?
rather than later?
e

early-musician-41645

11/05/2018, 7:57 PM
All I want to do is set the WARM_IP_TARGET for the CNI plugin: https://docs.aws.amazon.com/eks/latest/userguide/cni-env-vars.html Currently I do it by upgrading the DaemonSet using the default EKS aws-k8s-cni helm chart with a modified value to set the WARM_IP_TARGET. I.e. it's happening after cluster creation. If I could skip all that and just assign a value during cluster creation then that would be a better solution
I literally take a copy of this: https://github.com/aws/amazon-vpc-cni-k8s/blob/master/config/v1.1/aws-k8s-cni.yaml And I add a new env var for 
WARM_IP_TARGET
and deploy it after the cluster is created. I'm trying to add that chart as part of the Pulumi project but it's failing with no particular workaround I can find other than deploying it outside the Pulumi stack
m

microscopic-florist-22719

11/05/2018, 8:07 PM
to be a bit more concrete w.r.t. what I'm thinking here
I am currently teasing the EKS cluster, the worker pool, and the k8s resources out into separate components
so you could create the EKS cluster itself, then apply extra k8s resources (e.g. the cni chart referenced above), then create the worker pool, then deploy the dashboard
does that sound like it will work for you @early-musician-41645?
e

early-musician-41645

11/05/2018, 9:10 PM
It sounds like it could work, but as far as user experience it's not ideal. I'd rather just pass in a bunch of 
CNIArgs
to the 
new eks.Cluster
that I'm already using.
I'm using the 
eks.Cluster
to avoid getting into the weeds of cluster and worker group creation. There's just a small change I need for the CNI plugin that I want to orchestrate along with cluster creation
m

microscopic-florist-22719

11/05/2018, 9:21 PM
that's good feedback. just to make sure we're on the same page, here's the experience I'm working on:
Copy code
// create the EKS cluster and its other core resources
const core = new eks.Core(...);
// deploy additional k8s resources
// create the EKS worker pool
const workers = new eks.WorkerPool(...);
// deploy the k8s dashboard
const dashboard = new eks.Dashboard(...);
I will look more closely at the CNI bits and see if there's something higher-level we can do there
e

early-musician-41645

11/05/2018, 9:23 PM
I think it's a legit way to structure things, i.e. ask clients to be explicit and break down resource creation if there are customizations required. If that's the workflow then I'd expect to pass in some 
CNIPluginArgs
to one of those (the WorkerPool?)
As a workaround - I can 
kubectl apply -f aws-k8s-cni.yaml
and that solve the issue. Is there an equivalent to 
kubectl apply
from a local file I can use with Pulumi?
m

microscopic-florist-22719

11/05/2018, 9:24 PM
got it. my expectation with that approach was that you would set up the CNI bits manually before booting the workers (i.e. where it says 
// deploy additional k8s resources
), but from what you've described I think that we can do better here
e

early-musician-41645

11/05/2018, 9:24 PM
x-posted on the #general
m

microscopic-florist-22719

11/05/2018, 9:24 PM
Is there an equivalent to 
kubectl apply
from a local file I can use with Pulumi?
I'll defer to @creamy-potato-29402 on that. I don't believe that we have something with identical semantics, but he's the expert.
e

early-musician-41645

11/05/2018, 9:26 PM
Otherwise I'll just have to run a bunch of commands in one project, wait for it to finish and run my 
kubectl apply
, and then update a 2nd project
m

microscopic-florist-22719

11/05/2018, 9:27 PM
you could also shell out to 
kubectl
from within Pulumi
b

big-piano-35669

11/05/2018, 9:27 PM
Just to clarify so I answer correctly 🙂 Do you want to deploy a Kubernetes config file from within Pulumi? Or are you looking to deploy a Pulumi program with a 
pulumi up
command that works like 
apply -f $path_to_file
?
you could also shell out to 
kubectl
from within Pulumi
That shouldn't be necessary.
From a Pulumi program, you can say:
m

microscopic-florist-22719

11/05/2018, 9:28 PM
@big-piano-35669 IIUC it is currently necessary if the resource already exists
otherwise we'll try and fail to create it
e

early-musician-41645

11/05/2018, 9:29 PM
Ideally every resource is owned entirely within Pulumi. The CNI value overrides aren't implemented yet, so as a workaround I update the stack and then on the command line (i.e. outside Pulumi) I run 
kubectl apply
. I'd love to do with inside.
yeah, this is all due to the resource already existing as part of the 
new eks.Cluster
.
b

big-piano-35669

11/05/2018, 9:30 PM
Copy code
const k = new k8s.ConfigFile(
    "foo", { file: "config/foo.yml" });
to deploy a single Kubernetes YAML file, or
Copy code
const ks = new k8s.ConfigGroup(
    "foos", { files: "config/*.yaml" });
to deploy a collection of them.
e

early-musician-41645

11/05/2018, 9:30 PM
The SA in k8s is created but not tracked as a Pulumi resource
Copy code
const cni = new k8s.ConfigFile("aws-k8s-cni", { file: "aws-k8s-cni.yaml", });
Copy code
Diagnostics:
  pulumi:pulumi:Stack (eks-cluster-online-vnext-10az):
    error: Running program '/home/tsi.lan/eshamay/git/mustang/sdp-mustang-terraform/pulumi/eks-cluster' failed with an unhandled exception:
    error: TSError: ⨯ Unable to compile TypeScript:
    index.ts(89,21): error TS2339: Property 'ConfigFile' does not exist on type 'typeof import("/home/tsi.lan/eshamay/git/mustang/sdp-mustang-terraform/pulumi/eks-cluster/node_modules/@pulumi/kubernetes/index")'.

        at createTSError (/home/tsi.lan/eshamay/git/mustang/sdp-mustang-terraform/pulumi/eks-cluster/node_modules/ts-node/src/index.ts:261:12)
        at getOutput (/home/tsi.lan/eshamay/git/mustang/sdp-mustang-terraform/pulumi/eks-cluster/node_modules/ts-node/src/index.ts:367:40)
        at Object.compile (/home/tsi.lan/eshamay/git/mustang/sdp-mustang-terraform/pulumi/eks-cluster/node_modules/ts-node/src/index.ts:558:11)
        at Module.m._compile (/home/tsi.lan/eshamay/git/mustang/sdp-mustang-terraform/pulumi/eks-cluster/node_modules/ts-node/src/index.ts:439:43)
        at Module._extensions..js (module.js:586:10)
        at Object.require.extensions.(anonymous function) [as .ts] (/home/tsi.lan/eshamay/git/mustang/sdp-mustang-terraform/pulumi/eks-cluster/node_modules/ts-node/src/index.ts:442:12)
        at Module.load (module.js:494:32)
        at tryModuleLoad (module.js:453:12)
        at Function.Module._load (module.js:445:3)
        at Module.require (module.js:504:17)
    error: an unhandled error occurred: Program exited with non-zero exit code: 1
Is this in 0.16.1?
c

creamy-potato-29402

11/05/2018, 9:33 PM
@big-piano-35669 yeah that won’t work, because the resource already exists, I think.
e

early-musician-41645

11/05/2018, 9:34 PM
kubectl apply -f aws-k8s-cni.yaml
does work, and it reports things as 
unchanged
As long as Pulumi is calling 
kubectl
under the hood, I'd expect similar results
c

creamy-potato-29402

11/05/2018, 9:34 PM
the simplest hackiest thing to do is to shell out for now… I think the plan is that @microscopic-florist-22719 is going to expose more stuff so we have a cleaner answer to this? (I only skimmed the thread.)
Pulumi uses the Kubernetes Go client under the hood.
If you’re not going through the official declarative Pulumi API, these changes would not be reported in 
preview
Also you have to be careful — shelling out will execute that 
apply
every time you 
pulumi up
We really should just make the EKS SDK work for you. 🙂
m

microscopic-florist-22719

11/05/2018, 9:36 PM
Yes
e

early-musician-41645

11/05/2018, 9:36 PM
okay, agreed 😉
Do you know why I'd hit this error?
Copy code
error: TSError: ⨯ Unable to compile TypeScript:
    index.ts(89,21): error TS2339: Property 'ConfigFile' does not exist on type 'typeof import("/home/tsi.lan/eshamay/git/mustang/sdp-mustang-terraform/pulumi/eks-cluster/node_modules/@pulumi/kubernetes/index")'.
This is the offending code: 
const cni = new k8s.ConfigFile("aws-k8s-cni", { file: "aws-k8s-cni.yaml", });
c

creamy-potato-29402

11/05/2018, 9:39 PM
k8s
here should refer to 
@pulumi/kubernetes
the EKS package doesn’t contain that API.
e

early-musician-41645

11/05/2018, 9:41 PM
yeah I have this: 
import * as k8s from "@pulumi/kubernetes";
c

creamy-potato-29402

11/05/2018, 9:42 PM
k8s.yaml.ConfigFile
e

early-musician-41645

11/05/2018, 9:43 PM
okay, that just confirms I can't do that method either because the resources already exist... 😞 I'll have to shell out for now.
c

creamy-potato-29402

11/05/2018, 9:51 PM
@early-musician-41645 we’re conferring internally about how we can resolve this.
one second…
e

early-musician-41645

11/05/2018, 9:53 PM
I have a workaround for now, so it's not urgent. But will be once we're ready to send a few things out to other teams
c

creamy-potato-29402

11/05/2018, 9:54 PM
@early-musician-41645 it’s urgent to me. 🙂
e

early-musician-41645

11/05/2018, 9:54 PM
👍 thanks
c

creamy-potato-29402

11/05/2018, 10:00 PM
@early-musician-41645 @microscopic-florist-22719’s plan, I believe, is to solve the problem for you now; in the next-few-weeks timeframe we’re probably going to solve this with creation hooks: https://github.com/pulumi/pulumi/issues/1691
and the permanent solution is probably to support some notion of “overlays” in kubernetes at the very least. I’m filing an issue now and will follow up.
e

early-musician-41645

11/05/2018, 10:01 PM
awesome! Thanks for the effort on this. It's not mission critical at the moment, but I think that it's likely going to be a use-case that comes up in the future
b

big-piano-35669

11/05/2018, 10:05 PM
While you're here, I'm curious on one bit: Would you have expected Pulumi to automatically tolerate the existence of a resource, more like 
apply -f
? Or do you prefer an opt-in mechanism? Or just keep doing what we do now?
m

microscopic-florist-22719

11/05/2018, 10:13 PM
I’m also curious to know what sort of options you’d expect related to CNI: I’m not yet familiar with it, so I’m not sure what to expose.
e

early-musician-41645

11/05/2018, 10:13 PM
I expected that since Pulumi created the 
aws-nodes
ServiceAccount (i.e. via the EKS cluster and worker group) then it should tolerate the request to create it again, e.g. as an unchanged resource
very much like 
kubectl apply
would handle it. It would just report 
unchanged
.
@microscopic-florist-22719 These should probably be exposed in some form: https://docs.aws.amazon.com/eks/latest/userguide/cni-env-vars.html
👍 1
c

creamy-potato-29402

11/05/2018, 10:19 PM
I’ve written up some thoughts on how we’d allow for a more 
kubectl apply
-ish model here.
That’s the longer-term question I was talking about.
e

early-musician-41645

11/05/2018, 10:21 PM
I'm not sure I totally follow the wording on this:
the general problem of patching system resources that you didn't create.
The original issue arose because of the 
aws-nodes
SA. I did create the SA because I did a 
new eks.Cluster
. Does that fall into "resources that you didn't create"?
c

creamy-potato-29402

11/05/2018, 10:22 PM
Yes.
Basically, I’m saying that every kube-as-a-service provider gives you some number of things that are “part of the system”, like the dashboard, SAs, etc.
It is useful to be able to change those out-of-band.
e

early-musician-41645

11/05/2018, 10:24 PM
Okay, yes, agreed
c

creamy-potato-29402

11/05/2018, 10:25 PM
Ok, we have a short, medium, and long term solution. I believe @microscopic-florist-22719 is planning to deliver short-term in a day or two. Does this match up with what you’re expecting?
e

early-musician-41645

11/05/2018, 10:26 PM
yes, thanks a lot!
c

creamy-potato-29402

11/05/2018, 10:28 PM
great.
bother us if something else doesn’t work 🙂
Powered by