https://pulumi.com logo
#general
Title
# general
g

glamorous-thailand-23651

06/21/2019, 9:04 PM
hi folks, i’m trying to stand up an EKS cluster per some of the ‘getting started’ documentation for crosswalk. it seems some of the out-of-the-box k8s services are not starting due to no nodes being ready. my nodes are reporting ‘not ready’ due to the network not being initialized:
Copy code
$ kubectl describe nodes | grep 'KubeletNotReady' | head -n1
   Ready            False   Fri, 21 Jun 2019 13:57:30 -0700   Fri, 21 Jun 2019 13:50:29 -0700   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
to my eye, it seems to be same/similar issue as: https://github.com/pulumi/pulumi-kubernetes/issues/578 https://github.com/pulumi/pulumi-eks/issues/174 is there a known workaround?
the relevant bits of my index.ts are as follows:
Copy code
const zone = pulumi.output(aws.route53.getZone({name: config.require("apex-domain")}));

const vpc = new awsx.ec2.Vpc("vpc", {
    cidrBlock: "10.1.0.0/16",
    numberOfNatGateways: 2,
    numberOfAvailabilityZones: "all",
    tags: {"Name": "vpc"}
});

const allSubnetIds = vpc.privateSubnetIds.concat(vpc.publicSubnetIds);

// Create an EKS cluster with the default configuration.
const cluster = new eks.Cluster("k8s", {
    vpcId: vpc.id,
    subnetIds: allSubnetIds,
    nodeAssociatePublicIpAddress: false,
});
pulumi library versions:
Copy code
@pulumi/aws: 0.18.13
@pulumi/awsx: 0.18.6
@pulumi/docker: 0.17.0
@pulumi/eks: 0.18.8
@pulumi/kubernetes: 0.24.0
@pulumi/pulumi: 0.17.18
@pulumi/query: 0.3.0
l

lemon-spoon-91807

06/24/2019, 8:55 PM
@creamy-potato-29402 @breezy-hamburger-69619 can you help out here? Thanks!
g

glamorous-thailand-23651

06/24/2019, 8:56 PM
thanks @lemon-spoon-91807!
i’m running an
npm update
now after the
awsx
update announcement and going to try another
pulumi up
to see if i get different behavior.
seems same so far.
Copy code
1 Pods failed to schedule because: [Unschedulable] 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.
and the
kubectl describe nodes
output is similar to above.
b

breezy-hamburger-69619

06/24/2019, 9:00 PM
The nodes not available and the Node not being able to connect due to CNI issues are separate. The former means nodes are tainted that Pods dont have tolerances for. The latter is usually due version mismatches between control plane & workers, or, roles were not setup right for the nodegroup (assuming there is one).
Which error are you currently experiencing?
g

glamorous-thailand-23651

06/24/2019, 9:02 PM
both 🙂 my understanding was that the nodes are not ready because of the CNI issue, so can’t be scheduled. but i’m happy to be corrected
the
pulumi up
is about to fail after sufficient retries, i’ll grab the output ASAP.
i found the CNI message when trying to dig in on a possible cause why the nodes weren’t available.
b

breezy-hamburger-69619

06/24/2019, 9:03 PM
Hmm, I’m not tracking. By default, nodes are not tainted unless you supply taints to the NodeGroup specifically
g

glamorous-thailand-23651

06/24/2019, 9:04 PM
i haven’t done so. this is from a fresh stack
b

breezy-hamburger-69619

06/24/2019, 9:05 PM
Is there a full repro start to finish you can share?
g

glamorous-thailand-23651

06/24/2019, 9:06 PM
sure, let me open up my github repo — it’s quite simple, slightly modified from the ‘getting started’ guides
here’s the
Diagnostic
section from
pulumi up
that just failed:
Copy code
Diagnostics:
  kubernetes:apps:Deployment (kube-system/kubernetes-dashboard):
    error: Plan apply failed: 3 errors occurred:
    	* Timeout occurred for 'kubernetes-dashboard'
    	* Minimum number of live Pods was not attained
    	* 1 Pods failed to schedule because: [Unschedulable] 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.

  kubernetes:extensions:Deployment (kube-system/monitoring-influxdb):
    error: Plan apply failed: 3 errors occurred:
    	* Timeout occurred for 'monitoring-influxdb'
    	* Minimum number of Pods to consider the application live was not attained
    	* 1 Pods failed to schedule because: [Unschedulable] 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.

  kubernetes:core:Service (kube-system/heapster):
    error: Plan apply failed: 2 errors occurred:
    	* Timeout occurred for 'heapster'
    	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods

  kubernetes:core:Service (kube-system/monitoring-influxdb):
    error: Plan apply failed: 2 errors occurred:
    	* Timeout occurred for 'monitoring-influxdb'
    	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods

  kubernetes:core:Service (kube-system/kubernetes-dashboard):
    error: Plan apply failed: 2 errors occurred:
    	* Timeout occurred for 'kubernetes-dashboard'
    	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods

  kubernetes:extensions:Deployment (kube-system/heapster):
    error: Plan apply failed: 3 errors occurred:
    	* Timeout occurred for 'heapster'
    	* Minimum number of Pods to consider the application live was not attained
    	* 1 Pods failed to schedule because: [Unschedulable] 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.

  pulumi:pulumi:Stack (megadoomer-io-infra):
    error: update failed
let me know if there’s more output that would be helpful from the most recent run. figured this would be the most helpful.
b

breezy-hamburger-69619

06/24/2019, 9:09 PM
Thanks for that. For context, how is this repro connected to the linked issues (IAM role not supplied, or kube-dashboard not working) - or is this separate? I don’t see a reference to IAM or kube-dash in the repro
g

glamorous-thailand-23651

06/24/2019, 9:10 PM
it’s possible that it’s not, and if that’s the case i apologize. my interpretation was that it was a fresh stack where everything comes up except for these deployment/service resources and they still cannot come up after repeated
up
attempts
b

breezy-hamburger-69619

06/24/2019, 9:10 PM
correction: kube-dash is expected since its being deployed by default - duh, my mistake
what does
kubectl get nodes
return?
g

glamorous-thailand-23651

06/24/2019, 9:11 PM
Copy code
$ kubectl get nodes
NAME                                         STATUS     ROLES    AGE   VERSION
ip-10-1-129-147.us-west-2.compute.internal   NotReady   <none>   3d    v1.13.7-eks-c57ff8
ip-10-1-192-106.us-west-2.compute.internal   NotReady   <none>   3d    v1.13.7-eks-c57ff8
b

breezy-hamburger-69619

06/24/2019, 9:12 PM
now,
kubectl version
g

glamorous-thailand-23651

06/24/2019, 9:13 PM
Copy code
$ kubectl version
Client Version: <http://version.Info|version.Info>{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-20T04:49:16Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: <http://version.Info|version.Info>{Major:"1", Minor:"12+", GitVersion:"v1.12.6-eks-d69f1b", GitCommit:"d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7", GitTreeState:"clean", BuildDate:"2019-02-28T20:26:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
b

breezy-hamburger-69619

06/24/2019, 9:13 PM
there’s our problem
you’re on a 1.12 control plane, with nodes on 1.13
you cant have kubelets ahead of the control plane, thats what causes the CNI issue
and in turn bubbles up to no nodes available
g

glamorous-thailand-23651

06/24/2019, 9:14 PM
hm i see — i’m not sure how things got this way. is there something i can/should do within my pulumi stack?
FWIW the first time that i experienced this, i did a
pulumi destroy
and then another
pulumi up
to see if it was some kind of transient issue
b

breezy-hamburger-69619

06/24/2019, 9:14 PM
but this doesn’t make sense given your repro how you would up here, and given that we just patched this when 1.13 came out last week to match the workers against the server - see https://github.com/pulumi/pulumi-eks/pull/175
did you always have pulumi/eks v0.18.8 (which includes this PR ^)
?
g

glamorous-thailand-23651

06/24/2019, 9:16 PM
i wish i could give a sure answer. i’ve been doing a lot of playing with this, `destroy`ing and `up`ing a few times. i’m happy to
destroy
it again, sync + double check version numbers, then
up
again if that’s an expected path forward
b

breezy-hamburger-69619

06/24/2019, 9:17 PM
All good! What I meant was, were you always using v0.18.8 of the
pulumi/eks
package per
npm install
or did you update this from an older version?
g

glamorous-thailand-23651

06/24/2019, 9:18 PM
i see — yes, it’s quite possible as i did do an
npm update
at one point (and all my pulumi packages are set to
latest
) and got new versions. i didn’t pay much attention to what the exact version changes were
b

breezy-hamburger-69619

06/24/2019, 9:19 PM
I wonder if you hit this on an older version of
pulumi/eks
than v0.18.8. i’ll try re-creating your repro on v0.18.8 to see what that does
g

glamorous-thailand-23651

06/24/2019, 9:20 PM
ah, i did find a commit where i updated from 0.18.7 -> 0.18.8
b

breezy-hamburger-69619

06/24/2019, 9:20 PM
that would do it
this was broken on 0.18.7
g

glamorous-thailand-23651

06/24/2019, 9:21 PM
i see 🙂 sounds like we’re getting to the bottom of things
b

breezy-hamburger-69619

06/24/2019, 9:23 PM
I’m trying this repro on my end on a clean stack using
pulumi/eks
v0.18.8 and will report back
I suspect it’ll be fine
g

glamorous-thailand-23651

06/24/2019, 9:24 PM
ok, great. let me know if you need any more info from me. (ping me probably, in case i’m doing ‘real work’ in another slack)
👍🏼 1
and, FWIW, this is all playground for me; no data that needs retaining or stakeholders — so if the end move is to destroy/recreate that’s all fine with me
b

breezy-hamburger-69619

06/24/2019, 9:25 PM
👍🏼
heads up, per [1] and [2], heapster is broken out of the box, and we’re looking to deprecate the dashboard from the cluster definition, and instead document how to roll the helm chart [2], as kube-dashboard is just not getting enough support in the community to keep up w/ the updates. I personally always deploy clusters with
deployDashboard: false
1 - https://github.com/pulumi/pulumi-kubernetes/issues/600 2 - https://github.com/pulumi/pulumi-eks/issues/155#issuecomment-504096740
g

glamorous-thailand-23651

06/24/2019, 9:31 PM
ah, i see. i’ll add that option
b

breezy-hamburger-69619

06/24/2019, 9:46 PM
@glamorous-thailand-23651 i was able to run your repro error-free on v0.18.8 - all of my nodes & pods come up:
Copy code
metral@argon:~/megadoomer-io/stacks/infra$ kubectl get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-1-132-209.us-east-2.compute.internal   Ready    <none>   11m   v1.12.7
ip-10-1-169-248.us-east-2.compute.internal   Ready    <none>   11m   v1.12.7



metral@argon:~/megadoomer-io/stacks/infra$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY   STATUS    RESTARTS   AGE
kube-system   aws-node-2f2z8                          1/1     Running   0          11m
kube-system   aws-node-ztlld                          1/1     Running   0          11m
kube-system   coredns-65f768bbc8-5d4fx                1/1     Running   0          14m
kube-system   coredns-65f768bbc8-7jqq6                1/1     Running   0          14m
kube-system   heapster-684777c4cb-wvskb               1/1     Running   0          10m
kube-system   kube-proxy-kggzd                        1/1     Running   0          11m
kube-system   kube-proxy-nxnhr                        1/1     Running   0          11m
kube-system   kubernetes-dashboard-67d4c89764-rzd6m   1/1     Running   0          10m
kube-system   monitoring-influxdb-5c5bf4949d-sqxsd    1/1     Running   0          10m


metral@argon:~/megadoomer-io/stacks/infra$ kubectl version
Client Version: <http://version.Info|version.Info>{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"ab91afd7062d4240e95e51ac00a18bd58fddd365", GitTreeState:"clean", BuildDate:"2019-02-26T12:59:46Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: <http://version.Info|version.Info>{Major:"1", Minor:"12+", GitVersion:"v1.12.6-eks-d69f1b", GitCommit:"d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7", GitTreeState:"clean", BuildDate:"2019-02-28T20:26:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Please try on a clean stack using v0.18.8 - you should not hit it
g

glamorous-thailand-23651

06/24/2019, 9:48 PM
ok great - i probably would have ended up doing that at some point anyway, but glad to know the EKS destroy/create time won’t go to waste 🙂
b

breezy-hamburger-69619

06/24/2019, 9:48 PM
🙂
g

glamorous-thailand-23651

06/24/2019, 10:12 PM
@breezy-hamburger-69619 bringing up the new stack now. just want to confirm that with
deployDashboard: false
, no pods are deployed by default? in which case i should confirm success with
kubectl version ; kubectl get nodes
, and if everything looks good i should have no problem deploying other pods (via helm charts, etc) right?
and if that’s all correct i will gladly say thanks and wish you a good day 😄
b

breezy-hamburger-69619

06/24/2019, 10:14 PM
Only the kube dashboard and heapster pods won't come up with that flag enabled. You'll still see the required pods installed by EKS: core-dns, aws-node, kube-proxy
That said in my repro I did not set that flag and the dash and heapster do come up and run
Per issues, heapster doesn't show up in dash graphs, and we're considering depreciating the dash all together due to lack of community support
g

glamorous-thailand-23651

06/24/2019, 10:16 PM
ok- i’ll try it with that flag set to false for the moment, see what comes up after
aws:eks:Cluster                   k8s-eksCluster                    creating...
is done (i don’t currently see core-dns, aws-node, kube-proxy listed in the output yet) — and then give the dash+heapster a try in another iteration.
aye
b

breezy-hamburger-69619

06/24/2019, 10:16 PM
You will only see the pods in a
kubectl get pods --all-namespaces
, not in the Pulumi output
As we don't install them, eks does
g

glamorous-thailand-23651

06/24/2019, 10:17 PM
though, i don’t currently have a need for dash+heapster, just going with the defaults in most places unless i find a reason not to
1
roger that. new to me 🙂 we use hashicorp’s nomad at work
b

breezy-hamburger-69619

06/24/2019, 10:17 PM
All good. Let us know if you need anything else
g

glamorous-thailand-23651

06/24/2019, 10:18 PM
👍
tyvm for taking the time!
🎉 1