https://pulumi.com logo
Powered by
Title
d

dry-journalist-60579

02/13/2023, 4:03 PM
Soo I’ve been pulling my hair out over this. I wanted to build a stack to implement a set of password policies across all the accounts in my org. I set up the accounts using ControlTower so there are a standard set of Roles, etc. that come “out of the box.” I’m using an assumed role in the 
Pulumi.yaml
(
aws:assumeRole: '{"roleArn": "arn:aws:iam::xxx:role/service-role/AWSControlTowerAdmin"}'
). I wrote a stack to loop over 
organization = aws.organizations.get_organization()
=> 
organization.accounts
. Within the loop over the organizations, I then use 
role_arn = pulumi.Output.format(
        "arn:aws:iam::{0}:role/AWSControlTowerExecution", account.id
    )

    # Create Provider to assume role
    provider = aws.Provider(
        f"Provider: {account.name}",
        assume_role={
            "roleArn": role_arn,
        },
    )
We’re using SSO, so to run this I got my aws cli ready by: 
aws configure sso
and logging in. Finally, I run 
AWS_PROFILE=my-profile pulumi preview
and it worked… from my gitpod instance. When I try to run it locally with the same set up I get: 
Type                              Name                                                  Plan       Info
 +   pulumi:pulumi:Stack               aws-admin-stack-aws-admin                             create     
 +   ├─ aws:iam:AccountPasswordPolicy  AccountPasswordPolicy: management account             create     
 +   ├─ aws:iam:Policy                 AllowSSOAdminsToAssumeAWSControlTowerAdmin            create     
 +   ├─ pulumi:providers:aws           Provider: xxx                                         create     
 +   ├─ pulumi:providers:aws           Provider: xxx                                         create     
 +   ├─ pulumi:providers:aws           Provider: xxx                                         create     
 +   ├─ pulumi:providers:aws           Provider: xxx                                         create     
 +   ├─ pulumi:providers:aws           Provider: xxx                                         create     
 +   ├─ pulumi:providers:aws           Provider: xxx                                         create     
 +   ├─ pulumi:providers:aws           Provider: xxx                                         create     
 +   ├─ aws:iam:RolePolicyAttachment   AllowSSOAdminsToAssumeAWSControlTowerAdminAttachment  create     
     └─ aws:iam:AccountPasswordPolicy  AccountPasswordPolicy: xxx                                       1 error

error: unable to validate AWS credentials.
    Details: no valid credential sources for  found.
    
    Please see
    for more information about providing credentials.
    
    Error: failed to refresh cached credentials, operation error SSO: GetRoleCredentials, https response error StatusCode: 0, RequestID: , request send failed, Get "<https://portal.sso.us-east-1.amazonaws.com/federation/credentials?account_id=xxx&role_name=AWSAdministratorAccess>": dial tcp: lookup <http://portal.sso.us-east-1.amazonaws.com|portal.sso.us-east-1.amazonaws.com> on 192.168.1.1:53: read udp 192.168.1.104:61325->192.168.1.1:53: i/o timeout≈
Anyone have any clues?
m

melodic-tomato-39005

02/13/2023, 5:27 PM
Unfortunately, that might be a bug (see also). I’ll look into it today - apologies!
d

dry-journalist-60579

02/13/2023, 5:38 PM
Ooh, might there be a pulumi version difference causing this between my machines?
m

melodic-tomato-39005

02/13/2023, 5:39 PM
I don’t fully understand it yet but it’s possible.
d

dry-journalist-60579

02/13/2023, 5:40 PM
hmm 
v3.54.0
on both… so it’s not that
m

melodic-tomato-39005

02/13/2023, 5:40 PM
I think it’s a bug in the pulumi aws provider.
d

dry-journalist-60579

02/13/2023, 5:41 PM
ahh ok—well please let me know if there’s anything I can do to help! I do have a set up where it works in one case but not in another and I’m not entirely sure why
m

melodic-tomato-39005

02/13/2023, 5:41 PM
Is the code the same in both cases?
d

dry-journalist-60579

02/13/2023, 6:37 PM
Yeah, the code is identical
(happy to hop on a zoom or whatever if it’d be helpful! just let me know and I can find some time)
m

melodic-tomato-39005

02/13/2023, 10:39 PM
Hey @dry-journalist-60579, I dug into this and other authentication issues in aws a bit, but couldn’t repro the problem. I suspect there is some local state that messes with things on your machine, since it works in the clean gitpod environment. Could you double-check things like environment variables, default aws profile etc., if you haven’t already?
d

dry-journalist-60579

02/14/2023, 12:05 AM
I tried to ensure versions of python and pulumi and awscli are all the same… I’ll check 
env
~/.aws/config
files are identical
Oooh ok, wow, I just disabled my local littlesntich network filter and it was able to run
I can’t seem to figure out why having little snitch on causes this issue as I don’t have any restrictions on these hosts/ports
m

melodic-tomato-39005

02/14/2023, 3:11 AM
Interesting! It matches the i/o timeout of the error message. Thanks for updating us!
d

dry-journalist-60579

02/14/2023, 4:23 AM
No problem, sorry for the false alarm! I am wonder what the interaction is here though… I may have to email Little Snitch support because there doesn’t seem to be any preference related to this
None of the Provider Boolean arguments seem to affect the behavior either
BTW, I did notice another issue with SSO-based AWS cli profiles, similar to this issue with VS Code: https://github.com/aws/aws-toolkit-vscode/issues/3009 Newer versions of the CLI seem to follow a different format for how settings are stored in ~/.aws/config. Rather than just a 
[profile xxx]
section there is also a 
[sso-session xxx]
section. The way the cli sets this up just doesn’t seem to work with 
pulumi up
. Rather, I have to manually go in and make sure it’s in the “legacy” format. This is the error: 
Exception: invoke of aws:iam/getRole:getRole failed: invocation of aws:iam/getRole:getRole returned an error: unable to validate AWS credentials.
    Details: loading configuration: profile "<profile-name>" is configured to use SSO but is missing required configuration: sso_region, sso_start_url
E.g. this works: 
[profile <profile-name>]
sso_start_url = https://<subdomain>.<http://awsapps.com/start|awsapps.com/start>
sso_region = <region>
sso_account_id = <account-id>
sso_role_name = AWSAdmini<role-name>stratorAccess
region = <region>
output = json
and this doesn’t work: 
[profile <profile-name>]
sso_session = <session-name>
sso_account_id = <account-id>
sso_role_name = <role-name>
region = <region>
output = json

[sso-session <session-name>]
sso_start_url = https://<subdomain>.<http://awsapps.com/start|awsapps.com/start>
sso_region = <region>
sso_registration_scopes = sso:account:access
b

billowy-army-68599

02/14/2023, 2:36 PM
@dry-journalist-60579 we don’t support the sso-session configuration yet. See https://github.com/pulumi/pulumi-aws/issues/2272
d

dry-journalist-60579

02/14/2023, 3:29 PM
ah, thank you, @billowy-army-68599. For what it’s worth, editing the 
~/.aws/config
manually yields a configuration that works without needing something like https://github.com/jaxxstorm/aws-sso-creds
But that utility you built is nifty 🙂 I like it
eval $(aws-sso-creds export)
Does it support something like: 
AWS_PROFILE=my-profile aws-sso-creds pulumi up
b

billowy-army-68599

02/14/2023, 3:33 PM
it has a 
-p
flag - i haven’t considered creating an execution mechanism, but that’s an interesting idea
d

dry-journalist-60579

02/14/2023, 3:47 PM
might be nice to have it just be a one-line wrapper to avoid needing to export any ENV vars in the current terminal session
a

alert-cartoon-12389

04/26/2023, 11:43 PM
What can be the solution of this problem?
d

dry-journalist-60579

04/26/2023, 11:49 PM
Hmm what do you mean?
a

alert-cartoon-12389

04/26/2023, 11:51 PM
@dry-journalist-60579 I am having this issue also with s3 bucket. It gives the similar issue of no aws credentials found with pulumi version 3.64.0. Did you able to find the solution of this issue?
d

dry-journalist-60579

04/26/2023, 11:52 PM
What’s the error you’re getting?
a

alert-cartoon-12389

04/26/2023, 11:53 PM
error: Preview failed: unable to validate AWS credentials.
    Details: no valid credential sources for Pulumi AWS Classic found.
When I do this, everything works fine: 
AWS_PROFILE=my-profile pulumi up
But when I try to do pulumi up only it gives me this error
d

dry-journalist-60579

04/26/2023, 11:55 PM
Is it a problem that you need to specify your profile?
a

alert-cartoon-12389

04/26/2023, 11:57 PM
Yes
But how can I specify this in code, I am setting aws::profile in pulumi yaml file
d

dry-journalist-60579

04/26/2023, 11:59 PM
Hmm I have a bit of a different set up now. We’re using aws-so-util and awsume to manage our AWS credentials locally
So doing that I’m able to just run pulumi up
a

alert-cartoon-12389

04/27/2023, 12:00 AM
Are you using this package in your github pipelines also?
d

dry-journalist-60579

04/27/2023, 12:00 AM
I’m using Pulumi Deployments for CI/CD
And have a aws role the pulumi deployment executor assumes
3 Views
#awsJoin Slack
Powered by