https://pulumi.com logo
Powered by
Title
w

wide-account-87747

01/20/2023, 12:00 AM
Howdy, I’m getting a 403 when I 
pulumi up
at the preview stage: 
aws:lambda:Function (PROJECT-exec-role):
    error: Preview failed: refreshing urn:pulumi:dev::PROJECT::aws:lambda/function:Function::PROJECT-exec-role: 1 error occurred:
        * AccessDeniedException:
        status code: 403, request id: 649a5d8a-83ac-4194-82b9-0b11a1309de7
I’ve confirmed that my AWS CLI credentials are current and valid, and can perform the needed action (updating the IAM role) with these credentials manually; and this error reads more as a 403 from Pulumi, not AWS. I’ve tried 
pulumi logout && pulumi login
to no avail.
b

billowy-army-68599

01/20/2023, 12:21 AM
it’s definitely a 403 from AWS. You can verify that in CloudTrail
w

wide-account-87747

01/20/2023, 12:22 AM
Roger, cheers, will do more digging
Is it possible to get any more detail on what request is failing in the preview?
b

billowy-army-68599

01/20/2023, 4:04 AM
You can use something like iamlive to see what calls are made to the aws api
w

wide-account-87747

01/20/2023, 4:34 AM
Using iamlive I can’t see any calls being made when running 
pulumi preview
, despite receiving the above error. If I make any call using the AWS CLI it is immediately logged by iamlive.
b

billowy-army-68599

01/20/2023, 4:43 AM
I’ve definitely seen api calls via preview. It’s late here but I’ll send an example tomorrow
w

wide-account-87747

01/20/2023, 4:44 AM
Is it possible to verify what credentials pulumi is using? I have many credentials kicking around and it’s concievable that it isn’t respecting AWS_PROFILE
Cheers for the help
b

billowy-army-68599

01/20/2023, 5:08 AM
https://www.pulumi.com/registry/packages/aws/api-docs/getcalleridentity/
w

wide-account-87747

01/20/2023, 5:15 AM
As pulumi can’t get past the preview stage it can never export any outputs for that info to be readable.
Ok through cloudtrail I can confirm that for reasons unknown, pulumi is trying to update resources in the production account (though I have the dev stack set). Is there any suggestion as to why this might be happening?
The AWS_PROFILE is for our development account, and in the pulumi dashboard I can confirm that the dev stack lists ARNs for resources in the dev account; but CloudTrail is showing requests from my role in the development account to resources in the production account (note that the whole ARN is correct for the resources, it isn’t simply substituting the account ID)
User: arn:aws:sts::DEVELOPMENT:assumed-role/AWSReservedSSO_IMPLEMENTER_ROLE/Nathaniel.Munk@cloudwave.com.au is not authorized to perform: lambda:GetFunction on resource: arn:aws:lambda:ap-southeast-2:PRODUCTION:function:PROJECT-e6df74f because no resource-based policy allows the lambda:GetFunction action
b

billowy-army-68599

01/20/2023, 2:53 PM
Can you share your stack configuration and your code?
w

wide-account-87747

01/23/2023, 2:16 AM
Config: 
config:
  aws:allowedAccountIds:
    - "DEVELOPMENT"
  aws:profile: Sandbox1
  aws:region: ap-southeast-2
Code:
scratch_13.ts
b

billowy-army-68599

01/23/2023, 3:08 PM
That allowed account id isn’t correct, it should be an aws account ID
w

wide-account-87747

01/23/2023, 10:40 PM
It’s a replacement, in my actual config it is an AWS account ID.
You can assume that anywhere you see DEVELOPMENT, PRODUCTION, ROLE, or PROJECT; there are suitable valid substitutions in my code.
b

billowy-army-68599

01/23/2023, 11:30 PM
apologies, makes sense. What AWS account ID does 
sandbox
have?
this is almost certainly a misconfiguration
aws:profile: Sandbox1
w

wide-account-87747

01/23/2023, 11:31 PM
No, I’ve confirmed that the Sandbox1 profile has the DEVELOPMENT account ID configured correctly.
b

billowy-army-68599

01/23/2023, 11:31 PM
does this aws profile have an assume role in it? what about your environment variables?
w

wide-account-87747

01/23/2023, 11:32 PM
It’s a federated role with IAM Identity Centre. Env is set to this profile, but behaves in the same way when env is cleared.
b

billowy-army-68599

01/23/2023, 11:33 PM
somewhere in your configuration or your Pulumi program you’re targeting the wrong account. It’s hard to say where that is, but ultimately the Pulumi program is just trying to do what it’s told. 
arn:aws:lambda:ap-southeast-2:PRODUCTION:function:PROJECT-e6df74f
Perhaps this particular stack was already deployed to production once? If you check your stack state with 
pulumi stack export
you may see a reference to it
w

wide-account-87747

01/23/2023, 11:36 PM
pulumi stack export | grep PRODUCTION
is blank
Plenty of references to ARNs in DEVELOPMENT
b

billowy-army-68599

01/23/2023, 11:38 PM
okay, I’ not sure what else to suggest. Somewhere in your configuration you’re targeting the production account somehow
w

wide-account-87747

01/23/2023, 11:38 PM
There is a seperate prod stack that targets the PRODUCTION account, and deploys successfully.
Is there any way to diagnose why pulumi is trying to use the PRODUCTION ARNs in the dev stack?
to make super clear, i have changed AWS_PROFILE to the prod profile, changed pulumi stack to prod, and 
pulumi up -y
runs without issue.
b

billowy-army-68599

01/23/2023, 11:40 PM
pulumi uses the aws go sdk so you’d follow the same pattern. • verify the current user with 
aws sts get-caller-identity
• use something like IAM live to intercept the calls to the API https://github.com/iann0036/iamlive
w

wide-account-87747

01/23/2023, 11:43 PM
yep, and the result of 
aws sts get-caller-identity
is as expected, the DEVELOPMENT role and account; but cloudtrail shows calls made from that role to the PRODUCTION account resources
Specifically a GetFunction request, which is 403'd
Is there a way to reset the dev stack without tearing down the AWS resources?
b

billowy-army-68599

01/23/2023, 11:44 PM
you’d have to remove the stack configration and reimport all the resource
w

wide-account-87747

01/23/2023, 11:44 PM
And is there any logging I can do to try and find where pulumi is pulling the PROD ARNs from?
i’ve tried 
-v 9
(et. al.) but i was unable to find where it accesses the ARNs around the time it shows the calls to the provider
b

billowy-army-68599

01/23/2023, 11:46 PM
pulumi up -v 9 --logtostderr
w

wide-account-87747

01/23/2023, 11:46 PM
Honestly I probably can tear down all the resources if it means getting this back to baseline, but I suppose I’d need to do that manually through CF if pulumi is unable to target the correct stack; once that is done would deleting and recreating a stack using this AWS_PROFILE be enough?
b

billowy-army-68599

01/23/2023, 11:47 PM
you can delete the resources and run 
pulumi up -r
and it’ll recreate all the resources
w

wide-account-87747

01/23/2023, 11:47 PM
Roger
I’m going to PM you this stderr, I havent done any substitutions of ARNs and account details so please don’t post this anywhere public.
Appreciate your diligence here, will tear down and recreate and let you know how it goes.
Is it possible to remove the local stack configuration and redownload it from pulumi cloud? All ARNs are reported correctly in the portal for both stacks.
(and for 
pulumi stack export
but it’s worth a shot)
b

billowy-army-68599

01/23/2023, 11:53 PM
you can download a snapshot
w

wide-account-87747

01/24/2023, 12:07 AM
Any guidance on how do to this? I can’t see an equivalent to 
pulumi stack export
in the web UI, is there an API call to retrieve a specific checkpoint?
b

billowy-army-68599

01/24/2023, 12:08 AM
if you go to the last update in the UI, you’ll see this download box
w

wide-account-87747

01/24/2023, 12:09 AM
Ah I see, cheers.
16 Views
#awsJoin Slack
Powered by