Hey folks. Anyone else suddenly unable to even

preview

their EKS resources?

[...]

    kubernetes:core/v1:Service istio-loadbalancer-hostname  warning: configured Kubernetes cluster is unreachable: unable to load schema information from the API server: Get "<https://D01A49DEECB6E23C1A3454E1676679ED.gr7.us-east-2.eks.amazonaws.com/openapi/v2?timeout=32s>": getting credentials: exec: executable aws failed with exit code 255
    kubernetes:core/v1:Service istio-loadbalancer-hostname  error: Preview failed: failed to read resource state due to unreachable cluster. If the cluster was deleted, you can remove this resource from Pulumi state by rerunning the operation with the PULUMI_K8S_DELETE_UNREACHABLE environment variable set to "true"
    pulumi:pulumi:Stack resemble-clusters-aws-staging1 running error: preview failed
    kubernetes:core/v1:Service istio-loadbalancer-hostname  1 error; 1 warning
    pulumi:pulumi:Stack resemble-clusters-aws-staging1  1 error; 6 messages
Diagnostics:
  kubernetes:core/v1:Service (istio-loadbalancer-hostname):
    warning: configured Kubernetes cluster is unreachable: unable to load schema information from the API server: Get "<https://D01A49DEECB6E23C1A3454E1676679ED.gr7.us-east-2.eks.amazonaws.com/openapi/v2?timeout=32s>": getting credentials: exec: executable aws failed with exit code 255
    error: Preview failed: failed to read resource state due to unreachable cluster. If the cluster was deleted, you can remove this resource from Pulumi state by rerunning the operation with the PULUMI_K8S_DELETE_UNREACHABLE environment variable set to "true"

  [...]

  pulumi:pulumi:Stack ([...]):
    error: preview failed

    <botocore.awsrequest.AWSRequest object at 0x78fc7c333190>
    <botocore.awsrequest.AWSRequest object at 0x77d185d577d0>

It seems the

kubeconfig

produced by the EKS provider is incomplete!? That

kubeconfig

has the following (redacted):

[...]
        "exec": {
          "apiVersion": "<http://client.authentication.k8s.io/v1beta1|client.authentication.k8s.io/v1beta1>",
          "args": [
            "eks",
            "get-token",
            "--cluster-name",
            "staging1",
            "--role",
            "arn:aws:iam::123456789012:role/Administrator"
          ],
          "command": "aws",
          "env": [
            {
              "name": "KUBERNETES_EXEC_INFO",
              "value": "{\"apiVersion\": \"<http://client.authentication.k8s.io/v1beta1\|client.authentication.k8s.io/v1beta1\>"}"
            }
          ]
        }
[...]

When I run that myself, I indeed get the reported error:

$ aws eks get-token --cluster-name staging1 --role arn:aws:iam::123456789012:role/Administrator

<botocore.awsrequest.AWSRequest object at 0x72e04d3b0a10>

$ echo $?
255

But if I add the missing

--region

flag to the command, then I can it just fine:

aws eks get-token --cluster-name staging1 --role arn:aws:iam::123456789012:role/Administrator --region=us-east-2
{
    "kind": "ExecCredential",
    "apiVersion": "<http://client.authentication.k8s.io/v1beta1|client.authentication.k8s.io/v1beta1>",
    "spec": {},
    "status": {
        "expirationTimestamp": "2024-05-16T15:44:50Z",
        "token": "k8s-aws-v1.[...]"
    }
}

So why does Pulumi's EKS provider suddenly produce a

kubeconfig

that doesn't work!? And/or what happened to cause the

--region

flag to have gone missing?

Hey I just checked, and older provider versions didn't set the

--region

flag before, did you by any chance have a default region configured for the AWS CLI? That being said, I think it makes sense to include the region. Here's the feature request for that: https://github.com/pulumi/pulumi-eks/issues/1038 Please add your 👍 to the GitHub issue to help us prioritize this The linked issue also has a workaround for injecting the region using transformations: https://github.com/pulumi/pulumi-eks/issues/1038#issuecomment-2036131649