Thread
#general
    s

    strong-table-84168

    1 month ago
    Hey, everyone! Scratching my head on a weird issue and was wondering if maybe someone else had seen this before. I’m using pulumi to bootstrap a cluster on eks in order to test our agents during CI/CD. The eks cluster is successfully , created but for some reason the nodes get stuck in a not ready state. Trying to get the logs from any pod returns “Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)” … so its hard to debug what exactly is going on. I also can’t connect to underlying instance itself (ssh times out) so Im sort of flying blind. Nothing can get scheduled, because of the node issue. Seems to be related to kubelet, because all of the nodes have the following for all of their conditions
    NodeStatusUnknown  Kubelet stopped posting node status.
    Im also just following the standard tutorial from pulumi docs.
    c

    clever-rose-11123

    1 month ago
    Sounds like possibly IAM or Network. Maybe try get the status from the aws-node and kube-proxy pods:
    kubectl get pods -n kube-system -o wide
    As per https://aws.amazon.com/premiumsupport/knowledge-center/eks-node-status-ready/
    s

    strong-table-84168

    1 month ago
    Yeah they are all running and scheduled on a node, can’t get the logs from them though
    c

    clever-rose-11123

    1 month ago
    Smells like permissions error but I could be wrong
    s

    strong-table-84168

    1 month ago
    Yeah, that definitely seems plausible
    f

    flat-laptop-90489

    1 month ago
    Without kubelet logs and other bootstrapping logs from the nodes themselves, it will be very difficult to diagnose.
    But agreed, networking or permissions both seem plausible