https://pulumi.com logo
Powered by
l

late-nest-59850

06/22/2023, 12:45 PM
Hi all, I need some help with the kubernetes provider. I did a pulumi refresh, and it seemed to have broken my pulumi state. I also get tons of logs like this:
Copy code
TypeError: Cannot read properties of undefined (reading 'data')
        at /home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/cluster.ts:576:103
        at /home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/pulumi/output.js:250:35
        at Generator.next (<anonymous>)
        at /home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/pulumi/output.js:21:71
        at new Promise (<anonymous>)
        at __awaiter (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/pulumi/output.js:17:12)
        at applyHelperAsync (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/pulumi/output.js:229:12)
        at /home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/pulumi/output.js:183:65
        at processTicksAndRejections (node:internal/process/task_queues:95:5)
    TypeError: Cannot read properties of undefined (reading 'data')
        at /home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/cluster.ts:576:103
        at /home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/pulumi/output.js:250:35
        at Generator.next (<anonymous>)
        at /home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/pulumi/output.js:21:71
        at new Promise (<anonymous>)
        at __awaiter (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/pulumi/output.js:17:12)
        at applyHelperAsync (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/pulumi/output.js:229:12)
        at /home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/pulumi/output.js:183:65
        at processTicksAndRejections (node:internal/process/task_queues:95:5)

    unhandled rejection: CONTEXT(156): resource:devopsEksCluster-provider[pulumi:providers:kubernetes]
    STACK_TRACE:
    Error:
        at Object.debuggablePromise (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/kubernetes/node_modules/@pulumi/runtime/debuggable.ts:74:75)
        at Object.registerResource (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/kubernetes/node_modules/@pulumi/runtime/resource.ts:401:5)
        at new Resource (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/kubernetes/node_modules/@pulumi/resource.ts:423:13)
        at new CustomResource (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/kubernetes/node_modules/@pulumi/resource.ts:810:9)
        at new ProviderResource (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/kubernetes/node_modules/@pulumi/resource.ts:854:9)
        at new Provider (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/provider.ts:56:9)
        at new Cluster (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/cluster.ts:1461:25)
        at Object.<anonymous> (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/aws/eks/eks.ts:12:28)
        at Module._compile (node:internal/modules/cjs/loader:1254:14)
        at Module.m._compile (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/ts-node/src/index.ts:439:23)
    unhandled rejection: CONTEXT(156): resource:devopsEksCluster-provider[pulumi:providers:kubernetes]
    STACK_TRACE:
    Error:
        at Object.debuggablePromise (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/kubernetes/node_modules/@pulumi/runtime/debuggable.ts:74:75)
        at Object.registerResource (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/kubernetes/node_modules/@pulumi/runtime/resource.ts:401:5)
        at new Resource (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/kubernetes/node_modules/@pulumi/resource.ts:423:13)
        at new CustomResource (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/kubernetes/node_modules/@pulumi/resource.ts:810:9)
        at new ProviderResource (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/kubernetes/node_modules/@pulumi/resource.ts:854:9)
        at new Provider (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/provider.ts:56:9)
        at new Cluster (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/@pulumi/cluster.ts:1461:25)
        at Object.<anonymous> (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/aws/eks/eks.ts:12:28)
        at Module._compile (node:internal/modules/cjs/loader:1254:14)
        at Module.m._compile (/home/wernich/WebstormProjects/infrastructure/infrastructure/devops/shared/node_modules/ts-node/src/index.ts:439:23)
    unhandled rejection: CONTEXT(156): resource:devopsEksCluster-provider[pulumi:providers:kubernetes]
I'm not really sure what the cause could be. I seems there is some dependency issue between the k8s provider created by EKS and the infra running on that provider?
s

salmon-account-74572

06/22/2023, 3:47 PM
Yeah, the K8s provider created by EKS is used to create the resources on the EKS cluster itself, and if (for whatever reason) you can’t reach/connect/authenticate to the EKS cluster then you’ll run into issues when running commands like 
pulumi refresh
and others. Are you able to successfully use 
kubectl
to access the EKS cluster?
l

late-nest-59850

06/23/2023, 8:44 AM
The cluster version was updated outside of Pulumi, and I tried to pull those changes in using pulumi refresh. I can still access the cluster with normal IAM authentication myself. I think it's some kind of circular dependency, where the k8s provider created by EKS need to be updated to a newer version of k8s, but all those resources are using the old version of the provider. So it tried to edit them but it needs to update the provider first. I am completely lost as to how I can fix this at the moment.
s

salmon-account-74572

06/23/2023, 2:20 PM
Would you mind opening an issue on https://github.com/pulumi/pulumi? I think you might be able to solve this with a provider alias; check out the docs on aliases (https://www.pulumi.com/docs/concepts/options/aliases/). The idea is that you would alias the new provider with the old provider’s name. There was also a recent thread on provider aliases here in the Pulumi Community Slack; let me see if I can find that as well.
l

late-nest-59850

06/23/2023, 3:42 PM
I'll try again on Monday and update on any findings. If the alias doesn't solve it, I'll gladly open an issue.
I used the provider globally after defining it like this:
Copy code
export const devopsEksProvider = devopsEksCluster.provider;
What would be the best way to alias that? Create a new one with that one's kubeconfig and alias it?
s

salmon-account-74572

06/23/2023, 3:55 PM
It’s a tad more complicated than that. You’d need to look at the existing Pulumi state file (accessible via 
pulumi stack export > file.json
) to get the current provider information. Then, in your code, you can use the 
alias
resource option to the provider defined above with the information gained from the state file. A fair amount of this is covered in the alias docs. I also need to find you the other thread! Got pulled into a meeting, but let me look for that now.
l

late-nest-59850

07/11/2023, 8:16 AM
@salmon-account-74572 sorry for the long turnaround. I think I solved this part - for now. I am now facing what I assume is this issue, thus still unable to move forward and fully test the 
alias
resolution. I've updated the pulumi, aws, eks and kubernetes providers to the latest stable release.
s

salmon-account-74572

07/11/2023, 3:35 PM
It looks you’ve already commented on the issue; I would suggest (if possible) that you try to add more information (Pulumi version, provider versions, NodeJS version, etc.) to aid in finding a cause. No promises, but I’ll ping some folks internally to see if there is a workaround.
@late-nest-59850 Can you share the output of 
pulumi about
?
l

late-nest-59850

07/12/2023, 1:45 PM
@salmon-account-74572 thanks for the help and visibility. The about as follows, with the list of resources removed:
Copy code
CLI          
Version      3.74.0
Go Version   go1.20.5
Go Compiler  gc

Plugins
NAME        VERSION
aws         5.41.0
aws         5.31.0
awsx        1.0.2
docker      3.6.1
eks         1.0.2
kubernetes  3.30.1
nodejs      unknown

Host     
OS       debian
Version  bookworm/sid
Arch     x86_64

This project is written in nodejs: executable='/bin/node' version='v18.16.1'

Current Stack: superb/devops/shared

Found no pending operations associated with stack

Backend        
Name           <http://pulumi.com|pulumi.com>
URL            <https://app.pulumi.com/xx>
User           xx
Organizations  xx

Dependencies:
NAME                VERSION
@pulumi/pulumi      3.74.0
@types/node         14.18.53
@pulumi/aws         5.41.0
@pulumi/awsx        1.0.2
@pulumi/eks         1.0.2
@pulumi/kubernetes  3.30.1
I'll add it to the issue as well.
@salmon-account-74572 Any progress on this issue? It still hard blocks that entire stack I have.
Any @echoing-match-29901 that can look at this. It completely bricks an entire stack I have and looks like a legit bug
s

salmon-account-74572

08/21/2023, 1:38 PM
@late-nest-59850 I don’t have any updates, but I’ll raise the issue internally. I would also recommend you comment again on the associated GitHub issue.
l

late-nest-59850

08/21/2023, 1:39 PM
Thanks @salmon-account-74572 🙂 I'll comment once more, thanks for visibility
b

billowy-army-68599

08/21/2023, 2:46 PM
@late-nest-59850 do you have a repro/code for us to reproduce this?
l

late-nest-59850

08/22/2023, 8:00 AM
@billowy-army-68599 I'll work on creating a repo to reproduce this, I explained what I think happened in the thread, but I'll also try to reproduce to help where ever I can.
Unfortunately, I could not recreate the issue locally. I can see some work being done on the ticket to at least fix the null issue being a blocker, thanks for that! I'll keep trying, as I'm not really sure how it can be that the certificate authority is undefined at any time, since it clearly exists: Pulumi state file:
Copy code
{
  "urn": "urn:pulumi:shared::devops::eks:index:Cluster::devopsEksCluster",
  "custom": false,
  "type": "eks:index:Cluster",
  "outputs": {
    "eksCluster": {
      "4dabf18193072939515e22adb298388d": "5cf8f73096256a8f31e491e813e4eb8e",
      "id": "superb-eks-devops",
      "packageVersion": "",
      "urn": "urn:pulumi:shared::devops::eks:index:Cluster$aws:eks/cluster:Cluster::devopsEksCluster-eksCluster"
    },
    "kubeconfig": {
      "apiVersion": "v1",
      "clusters": [
        {
          "cluster": {
            "certificate-authority-data": "xxx",
            "server": "<https://xxx.gr7.eu-central-1.eks.amazonaws.com>"
          },
          "name": "kubernetes"
        }
      ],
      "contexts": [
        {
          "context": {
            "cluster": "kubernetes",
            "user": "aws"
          },
          "name": "aws"
        }
      ],
      "current-context": "aws",
      "kind": "Config",
      "users": [
        {
          "name": "aws",
          "user": {
            "exec": {
              "apiVersion": "<http://client.authentication.k8s.io/v1beta1|client.authentication.k8s.io/v1beta1>",
              "args": [
                "eks",
                "get-token",
                "--cluster-name",
                "superb-eks-devops",
                "--role",
                "arn:aws:iam::xx:role/xx"
              ],
              "command": "aws",
              "env": [
                {
                  "name": "KUBERNETES_EXEC_INFO",
                  "value": "{\"apiVersion\": \"<http://client.authentication.k8s.io/v1beta1\|client.authentication.k8s.io/v1beta1\>"}"
                }
              ]
            }
          }
        }
      ]
    }
  },
  "parent": "urn:pulumi:shared::devops::pulumi:pulumi:Stack::devops-shared"
}
Is there any additional info I can provide?
Powered by