https://pulumi.com logo
Join Slack
Powered by
Heyo, anyone know how to fix issues with `pulumi u...
# general
g
Heyo, anyone know how to fix issues with 
pulumi up
in github actions? Works fine locally for me, but it just hangs forever until it times out in github actions:
Copy code
Run pulumi/actions@v6
  with:
    command: up
    stack-name: prod
    work-dir: infrastructure/pulumi
    comment-on-pr: false
    comment-on-summary: false
    github-token: ***
    expect-no-changes: false
    diff: false
    target-dependents: false
    refresh: false
    upsert: false
    remove: false
    edit-pr-comment: true
    color: auto
    exclude-protected: false
    suppress-outputs: false
    suppress-progress: false
    always-include-summary: false
    continue-on-error: false
  env:
    AWS_REGION: us-east-1
    ECR_REPOSITORY: lekondo/denim
    AWS_DEFAULT_REGION: us-east-1
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***
    AWS_SESSION_TOKEN: ***
    IMAGE_TAG: <redacted>
    PULUMI_ACCESS_TOKEN: ***
    AWS_ACCOUNT_ID: ***
    APP_ENV: prod
    ALLOWED_ORIGINS: <redacted>
    AUTH0_DOMAIN: <redacted>
    AUTH0_AUDIENCE: <redacted>
    AUTH0_TENANT_DOMAIN: <redacted>
    RUNPOD_API_URL: <redacted>
    VERBOSE_LOGGING: false
    DOMAIN_NAME: <redacted>
    CLOUDFLARE_API_TOKEN: ***
Configured range: 
/usr/local/bin/pulumi version
v3.163.0
warning: A new version of Pulumi is available. To upgrade from version '3.163.0' to '3.165.0', visit <https://pulumi.com/docs/install/> for manual instructions and release notes.
Pulumi version 3.163.0 is already installed on this machine. Skipping download
Logging into the Pulumi Cloud backend.
pulumi up on prod
  Updating (prod)
  
  
  View Live: <https://app.pulumi.com/mover/lekondo-denim/prod/updates/17>
  
  
  @ Updating....
  .
  .
  .
  .
  .
  .
  .
  .
  .
  .
  .
  .
  .
  .
  .
  .
  
  Downloading plugin cloudflare-6.1.1: starting
  
  Downloading plugin aws-6.66.3: starting
  
  @ Updating....
  
  Downloading plugin cloudflare-6.1.1: done
  
  Installing plugin cloudflare-6.1.1: starting
  
  Installing plugin cloudflare-6.1.1: done
  
  @ Updating....
  .
  
  Downloading plugin aws-6.66.3: done
  
  Installing plugin aws-6.66.3: starting
  
  @ Updating....
  .
  .
  
  Installing plugin aws-6.66.3: done
  
  
  
  @ Updating....
  .
  .
  .
  .
  .
  .
It does like 500 lines of periods, and the pulumi cloud update just fails with 0 logs
My action yaml:
Copy code
- name: Run Pulumi Up
        uses: pulumi/actions@v6 # Use the action directly
        id: pulumi-up
        with:
          command: up # Run pulumi up (action handles non-interactive)
          stack-name: prod # Specify the stack name
          work-dir: infrastructure/pulumi # Specify the working directory
          pulumi-version: latest # Ensure latest Pulumi CLI is used
          # Pulumi action automatically exports stack outputs
        env:
          PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
        <lots of other env vars>
m
hi @gray-jewelry-75640, thanks for reporting. let me check with engineering.
m
👋🏻 I just ran into this problem myself. Did you ever figure it out? Did we hear back from engineers?
m
Any chance for a link to the workflow in GitHub?
m
It is sadly on a private repository that I can't share publicly. I have narrowed this down to missing IAM policies.
m
So it works now for you?
m
I am getting progress, but I think any time Pulumi runs into incorrect permissions, it prints the error and then continues indefinitely.
m
ok that is good feedback.
m
actually, I can share some more interesting things, just give me a moment
m
Yes please, additionally always great to create an issue in GH with this findings.
m
logs.txt
logs.txt
got this from the failed staging build
m
private EKS? 
Error: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:43247: connect: connection refused
m
happens straight after:
Copy code
@ updating....
  .
  .
   +  aws:iam:OpenIdConnectProvider app-services-eks-cluster-oidc-provider creating (6s) error:   sdk-v2/provider2.go:572: sdk.helper_schema: reading IAM OIDC Provider (arn:aws:iam::999999999999:oidc-provider/oidc.eks.ap-southeast-2.amazonaws.com/id/ABCDEF1234567890987654321FEDCBA0): operation error IAM: GetOpenIDConnectProvider, https response error StatusCode: 403, RequestID: 2bc758da-2723-4c53-842f-6a4e5a0d6fff, api error AccessDenied: User: arn:aws:sts::122610496939:assumed-role/Pulumi-Deployment-Role/GitHubActions is not authorized to perform: iam:GetOpenIDConnectProvider on resource: arn:aws:iam::999999999999:oidc-provider/oidc.eks.ap-southeast-2.amazonaws.com/id/ABCDEF1234567890987654321FEDCBA0 because no identity-based policy allows the iam:GetOpenIDConnectProvider action: provider=aws@6.83.0
   +  aws:iam:OpenIdConnectProvider app-services-eks-cluster-oidc-provider creating (6s) error: 1 error occurred:
   +  aws:iam:OpenIdConnectProvider app-services-eks-cluster-oidc-provider **creating failed** error: 1 error occurred:
  @ updating....
  .
  .
  .
  .
I am going to fix the permission issue right now, but it will run into the same problem on the subsequent one I believe (I am in the process of figuring out all the IAM things it needs)
I am trying to deploy an EKS cluster, and "private" in there simply means the node group that is our private VPC subnet.
We have some in public for internet-facing workloads and some in private for internal things that don't need external access.
Alright, so I am coming back to report that it indeed happens when Pulumi tries to perform some operation on EKS, but fails due to insufficient permissions. I had everything working recently, then I decided to add an access policy association but didn't have the 
eks:AssociateAccessPolicy
action allowed. It hang again after reporting that I don't have permissions. I would expect Pulumi to fail and bail out, but it doesn't.
3 Views
Open in Slack
PreviousNext
Powered by