Hey, everyone! Scratching my head on a weird issue and was wondering if maybe someone else had seen this before. I’m using pulumi to bootstrap a cluster on eks in order to test our agents during CI/CD. The eks cluster is successfully , created but for some reason the nodes get stuck in a not ready state. Trying to get the logs from any pod returns “Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)” … so its hard to debug what exactly is going on. I also can’t connect to underlying instance itself (ssh times out) so Im sort of flying blind. Nothing can get scheduled, because of the node issue. Seems to be related to kubelet, because all of the nodes have the following for all of their conditions

NodeStatusUnknown  Kubelet stopped posting node status.

Im also just following the standard tutorial from pulumi docs.

Sounds like possibly IAM or Network. Maybe try get the status from the aws-node and kube-proxy pods:

kubectl get pods -n kube-system -o wide

As per https://aws.amazon.com/premiumsupport/knowledge-center/eks-node-status-ready/

Yeah they are all running and scheduled on a node, can’t get the logs from them though

Smells like permissions error but I could be wrong

Yeah, that definitely seems plausible

Without kubelet logs and other bootstrapping logs from the nodes themselves, it will be very difficult to diagnose.