https://pulumi.com logo
Join Slack
Powered by
Howdy, I’m getting a 403 when I `pulumi up` at the...
# aws
w
Howdy, I’m getting a 403 when I 
pulumi up
at the preview stage:
Copy code
aws:lambda:Function (PROJECT-exec-role):
    error: Preview failed: refreshing urn:pulumi:dev::PROJECT::aws:lambda/function:Function::PROJECT-exec-role: 1 error occurred:
        * AccessDeniedException:
        status code: 403, request id: 649a5d8a-83ac-4194-82b9-0b11a1309de7
I’ve confirmed that my AWS CLI credentials are current and valid, and can perform the needed action (updating the IAM role) with these credentials manually; and this error reads more as a 403 from Pulumi, not AWS. I’ve tried 
pulumi logout && pulumi login
to no avail.
b
it’s definitely a 403 from AWS. You can verify that in CloudTrail
w
Roger, cheers, will do more digging
Is it possible to get any more detail on what request is failing in the preview?
b
You can use something like iamlive to see what calls are made to the aws api
w
Using iamlive I can’t see any calls being made when running 
pulumi preview
, despite receiving the above error. If I make any call using the AWS CLI it is immediately logged by iamlive.
b
I’ve definitely seen api calls via preview. It’s late here but I’ll send an example tomorrow
w
Is it possible to verify what credentials pulumi is using? I have many credentials kicking around and it’s concievable that it isn’t respecting AWS_PROFILE
Cheers for the help
b
w
As pulumi can’t get past the preview stage it can never export any outputs for that info to be readable.
Ok through cloudtrail I can confirm that for reasons unknown, pulumi is trying to update resources in the production account (though I have the dev stack set). Is there any suggestion as to why this might be happening?
The AWS_PROFILE is for our development account, and in the pulumi dashboard I can confirm that the dev stack lists ARNs for resources in the dev account; but CloudTrail is showing requests from my role in the development account to resources in the production account (note that the whole ARN is correct for the resources, it isn’t simply substituting the account ID)
Copy code
User: arn:aws:sts::DEVELOPMENT:assumed-role/AWSReservedSSO_IMPLEMENTER_ROLE/Nathaniel.Munk@cloudwave.com.au is not authorized to perform: lambda:GetFunction on resource: arn:aws:lambda:ap-southeast-2:PRODUCTION:function:PROJECT-e6df74f because no resource-based policy allows the lambda:GetFunction action
b
Can you share your stack configuration and your code?
w
Config:
Copy code
config:
  aws:allowedAccountIds:
    - "DEVELOPMENT"
  aws:profile: Sandbox1
  aws:region: ap-southeast-2
b
That allowed account id isn’t correct, it should be an aws account ID
w
It’s a replacement, in my actual config it is an AWS account ID.
You can assume that anywhere you see DEVELOPMENT, PRODUCTION, ROLE, or PROJECT; there are suitable valid substitutions in my code.
b
apologies, makes sense. What AWS account ID does 
sandbox
have?
this is almost certainly a misconfiguration
Copy code
aws:profile: Sandbox1
w
No, I’ve confirmed that the Sandbox1 profile has the DEVELOPMENT account ID configured correctly.
b
does this aws profile have an assume role in it? what about your environment variables?
w
It’s a federated role with IAM Identity Centre. Env is set to this profile, but behaves in the same way when env is cleared.
b
somewhere in your configuration or your Pulumi program you’re targeting the wrong account. It’s hard to say where that is, but ultimately the Pulumi program is just trying to do what it’s told.
Copy code
arn:aws:lambda:ap-southeast-2:PRODUCTION:function:PROJECT-e6df74f
Perhaps this particular stack was already deployed to production once? If you check your stack state with 
pulumi stack export
you may see a reference to it
w
pulumi stack export | grep PRODUCTION
is blank
Plenty of references to ARNs in DEVELOPMENT
b
okay, I’ not sure what else to suggest. Somewhere in your configuration you’re targeting the production account somehow
w
There is a seperate prod stack that targets the PRODUCTION account, and deploys successfully.
Is there any way to diagnose why pulumi is trying to use the PRODUCTION ARNs in the dev stack?
to make super clear, i have changed AWS_PROFILE to the prod profile, changed pulumi stack to prod, and 
pulumi up -y
runs without issue.
b
pulumi uses the aws go sdk so you’d follow the same pattern. • verify the current user with 
aws sts get-caller-identity
• use something like IAM live to intercept the calls to the API https://github.com/iann0036/iamlive
w
yep, and the result of 
aws sts get-caller-identity
is as expected, the DEVELOPMENT role and account; but cloudtrail shows calls made from that role to the PRODUCTION account resources
Specifically a GetFunction request, which is 403'd
Is there a way to reset the dev stack without tearing down the AWS resources?
b
you’d have to remove the stack configration and reimport all the resource
w
And is there any logging I can do to try and find where pulumi is pulling the PROD ARNs from?
i’ve tried 
-v 9
(et. al.) but i was unable to find where it accesses the ARNs around the time it shows the calls to the provider
b
pulumi up -v 9 --logtostderr
w
Honestly I probably can tear down all the resources if it means getting this back to baseline, but I suppose I’d need to do that manually through CF if pulumi is unable to target the correct stack; once that is done would deleting and recreating a stack using this AWS_PROFILE be enough?
b
you can delete the resources and run 
pulumi up -r
and it’ll recreate all the resources
w
Roger
I’m going to PM you this stderr, I havent done any substitutions of ARNs and account details so please don’t post this anywhere public.
Appreciate your diligence here, will tear down and recreate and let you know how it goes.
Is it possible to remove the local stack configuration and redownload it from pulumi cloud? All ARNs are reported correctly in the portal for both stacks.
(and for 
pulumi stack export
but it’s worth a shot)
b
you can download a snapshot
w
Any guidance on how do to this? I can’t see an equivalent to 
pulumi stack export
in the web UI, is there an API call to retrieve a specific checkpoint?
b
if you go to the last update in the UI, you’ll see this download box
w
Ah I see, cheers.
251 Views
Open in Slack
PreviousNext
Powered by