https://pulumi.com logo
#aws
Title
# aws
a

astonishing-journalist-77684

12/01/2023, 9:14 PM
hello friends - I have stumbled upon an issue in the
pulumi-aws
driver we rely on. The symptoms match a number of open and closed issues on the repo, so I hope the question I have makes sense. When using SSO,
pulumi preview
and
pulumi up
both give the same error, consistently:
Copy code
Error: failed to refresh cached credentials, the SSO session has expired or is invalid: open /Users/[redacted]/.aws/sso/cache/1f00d08c5e62b5eaa7523500a301bc70997c42d9.json: no such file or directory
It's always the same hash, even across days, and no matter how many times I've renewed my AWS credentials. What I have been able to dig up through investigation of the aws sdk and the
pulumi-aws
repository is that at some point, pulumi had duplicated whatever hashing approach (sha-1?) that the aws sdk was performing at the time to generate new cache file names so that you could pull the latest cache file. Unfortunately, aws's approach has changed over time and now
pulumi-aws
generates an erroneous hash which sends it looking for the same non-existent file. The work-arounds I've found from other folks reporting this issue is to symlink the latest cache file to the file pulumi expects. This is irritating because I have to first run pulumi preview to get the error in order to know what hash it's trying to use for the file. I would like to add to my shell script that refreshes all my credentials each day that symlinks the hash pulumi expects to the latest cache file after I've refreshed the login. The catch is, I can't figure out where exactly you all are passing things in my environment into the hash function in order to determine that filename. Can someone here help me determine the approach so we can get a work around? Thank you in advance for reading the long-winded explanation 😅
b

billowy-army-68599

12/01/2023, 9:18 PM
@astonishing-journalist-77684 I think opening an issue on pulumi-aws with links to the upsream issues is the best bet
a

astonishing-journalist-77684

12/01/2023, 9:18 PM
hi Jaxx - there are several open (and closed) issues reporting this problem.
I figured instead of just opening another one, I could be proactive and automate a local work-around
and while I could run
pulumi preview
and wait for it to dump the error out and parse that looking for the missing file name, I thought it would be easier if I just asked how that hash is generated
b

billowy-army-68599

12/01/2023, 9:21 PM
the underlying AWS provider uses the aws go sdk to authenticate, i do not believe we generate any hashes or manipulate aws credentials in any way
it just follows the credential chain
what lead you to believe we’re changing the credential file?
a

astonishing-journalist-77684

12/01/2023, 9:21 PM
it's reading the cached credential file that aws stores after an sso login
the fix I mentioned is to symlink what pulumi thinks the file should be to the actual latest cache file
but aws is not giving pulumi that hash
b

billowy-army-68599

12/01/2023, 9:23 PM
can you link me to some of the issues you’re referring to?
it’s reading the cached credential file that aws stores after an sso login
if it’s doing this, again, the issue is related directly to the aws GO sdk. there’s nothing special in the provider that does this to my knowledge
also: are you using SSO sessions? how is your SSO profile configured
a

astonishing-journalist-77684

12/01/2023, 9:26 PM
sure thing, this is the latest: https://github.com/pulumi/pulumi-aws/issues/3023
b

billowy-army-68599

12/01/2023, 9:27 PM
are you using an S3 backend?
a

astonishing-journalist-77684

12/01/2023, 9:28 PM
the aws config is set to use an sso session, yes
I'll have to find out if we are, I'm in a new role and as I was trying to get pulumi working, I ran into this. Anyone on the team who moved to the latest version is also running into the issue now but were not before
b

billowy-army-68599

12/01/2023, 9:29 PM
at what stage does the error throw? during
pulumi login
or
pulumi up
?
a

astonishing-journalist-77684

12/01/2023, 9:29 PM
pulumi preview or pulumi up
basically, when it tries to read from aws
with the symlink in place, it will work for a day
but when I have to re-auth with AWS, a new cache file is created and then I have to delete and recreate the symlink pointing to a new file
b

billowy-army-68599

12/01/2023, 9:31 PM
if you look in your aws config, does the
sso_start_url
have a
#
in it?
a

astonishing-journalist-77684

12/01/2023, 9:31 PM
yes
b

billowy-army-68599

12/01/2023, 9:31 PM
remove it, make sure the AWS CLI is at the latest version, and I would bet money that solves the issue 🙂
a

astonishing-journalist-77684

12/01/2023, 9:31 PM
I'll give it a try now, seems like an easy enough thing 🙂
b

billowy-army-68599

12/01/2023, 9:33 PM
oh, make sure you reauth as well, once you’ve modified your start url. You might want to clear your sso cache
as an aside, I wrote a tool to automate the retrieval of temp credentials that I use every day: https://github.com/jaxxstorm/aws-sso-creds
a

astonishing-journalist-77684

12/01/2023, 9:35 PM
removing the hashmark from the end of that url broke the aws sso login
b

billowy-army-68599

12/01/2023, 9:36 PM
does it look like this?
Copy code
sso_start_url = <https://lbrlabs.awsapps.com/start>
a

astonishing-journalist-77684

12/01/2023, 9:36 PM
yes, after I removed the
#
at the end
b

billowy-army-68599

12/01/2023, 9:37 PM
what error do you get?
a

astonishing-journalist-77684

12/01/2023, 9:37 PM
Copy code
Missing the following required SSO configuration values: sso_start_url, sso_region. To make sure this profile is properly configured to use SSO, please run: aws configure sso
b

billowy-army-68599

12/01/2023, 9:37 PM
what version of the aws cli are you using?
a

astonishing-journalist-77684

12/01/2023, 9:38 PM
aws-cli/2.13.37 Python/3.11.6 Darwin/23.1.0
b

billowy-army-68599

12/01/2023, 9:39 PM
can you share your redacted
.aws/config
file?
a

astonishing-journalist-77684

12/01/2023, 9:41 PM
Copy code
[default]
region = [region]
output = json
[profile myprofile]
sso_session = myprofile
sso_account_id = [sso-account-id]
sso_role_name = [sso-role]
sso_region = [sso-region]
sso_start_url = [sso-start-url]
sso_registration_scopes = sso:account:access
region = [region]
output = json
[sso-session myprofile]
sso_start_url = [sso-start-url]
sso_region = [sso-region]
sso_registration_scopes = sso:account:access
b

billowy-army-68599

12/01/2023, 9:47 PM
that doesn’t look correct to me, that looks like a cross between a legacy profile and an sso session profile. Here’s how mine looks, and it works:
Copy code
[sso-session personal]
sso_region = us-west-2
sso_start_url = <https://lbrlabs.awsapps.com/start>

[profile personal-development]
sso_session = personal
output = json
region = us-west-2
sso_account_id = <x>
sso_role_name = AWSAdministratorAccess

[profile personal-management]
sso_session = personal
output = json
region = us-west-2
sso_account_id = <x>
sso_role_name = AWSAdministratorAccess
I’d choose to use sso-session and configure it correctly, or use a legacy profile (where you copy the start url and region to every profile)
a

astonishing-journalist-77684

12/01/2023, 9:48 PM
ok, I'll see if I can make edits to get this to work with your format and report back
ok, this is the config now:
Copy code
[sso-session dev]
sso_region = [sso-region]
sso_start_url = [sso-start-url]

[profile profile-name]
sso_session = dev
output = json
region = [profile-region]
sso_account_id = [sso-account-id]
sso_role_name = [sso-role-name]
the aws sso login works now, but, I get a different error from pulumi:
Copy code
aws:kms:Key (cluster-key):
    error: unable to validate AWS credentials.
    Details: loading configuration: profile "[profile-name]" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

    Make sure you have:

     	 • Set your AWS region, e.g. `pulumi config set aws:region us-west-2`
     	 • Configured your AWS credentials as per <https://pulumi.io/install/aws.html>
     	 You can also set these via cli using `aws configure`.

  pulumi:pulumi:Stack ([stack-name]):
    error: Error: invocation of aws:index/getPartition:getPartition returned an error: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
        at Object.callback (/Users/alex/git/rownd/infrastructure/node_modules/@pulumi/runtime/invoke.ts:159:33)
        at Object.onReceiveStatus (/Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/client.ts:338:26)
        at Object.onReceiveStatus (/Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
        at Object.onReceiveStatus (/Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
        at /Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24
        at processTicksAndRejections (node:internal/process/task_queues:77:11)
    error: Error: invocation of aws:index/getRegion:getRegion returned an error: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
        at Object.callback (/Users/alex/git/rownd/infrastructure/node_modules/@pulumi/runtime/invoke.ts:159:33)
        at Object.onReceiveStatus (/Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/client.ts:338:26)
        at Object.onReceiveStatus (/Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
        at Object.onReceiveStatus (/Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
        at /Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24
        at processTicksAndRejections (node:internal/process/task_queues:77:11)
b

billowy-army-68599

12/01/2023, 10:02 PM
which version of the aws provider are you using?
a

astonishing-journalist-77684

12/01/2023, 10:02 PM
I got stuck in this with the more sparse config the other day and we added the fields that my colleagues had until we got it working (using the symlink work-around mentioned before)
by that, you mean on the server?
b

billowy-army-68599

12/01/2023, 10:03 PM
no, the aws provider is configured in your stack. you can retrieve it by running
pulumi about
I suspect you’re on an outdated version of the pulumi aws provider with an embedded aws sdk go version that doesn’t actually support sso-session
a

astonishing-journalist-77684

12/01/2023, 10:04 PM
Copy code
Dependencies:
NAME                VERSION
@pulumi/aws-native  0.40.2
@pulumi/kubernetes  3.22.1
@pulumi/pulumi      3.46.1
@pulumi/cloudflare  4.12.1
@pulumi/eks         0.42.7
@pulumi/gitlab      4.9.0
@types/node         14.18.33
simple-git          2.48.0
@aws-cdk/aws-ec2    1.180.0
@pulumi/aws         5.21.0
@pulumi/awsx        0.40.1
b

billowy-army-68599

12/01/2023, 10:08 PM
yep, that’s quite old. you’ll need to either: • update your aws provider to use an aws sdk for go that allows using sso-session • use a legacy profile setup, where you set the start_url and sso_region in every profile
a

astonishing-journalist-77684

12/01/2023, 10:08 PM
right, which is what I had (bullet 2)
but then we're back to the cache file bug
b

billowy-army-68599

12/01/2023, 10:09 PM
you were still setting the sso_session property:
Copy code
[profile myprofile]
sso_session = myprofile # here
sso_account_id = [sso-account-id]
sso_role_name = [sso-role]
sso_region = [sso-region]
sso_start_url = [sso-start-url]
sso_registration_scopes = sso:account:access
region = [region]
output = json
a

astonishing-journalist-77684

12/01/2023, 10:09 PM
so - and this is where I need to read more docs - to update the dependencies, that would involve updating our stack definition to get to the latest?
b

billowy-army-68599

12/01/2023, 10:11 PM
yep,
npm install @pulumi/aws --update
The long and short of all this really is that there are bugs in sso-session and sso profile management and it’s finicky. as i mentioned before, this is the main upstream issue: https://github.com/aws/aws-sdk-go-v2/issues/2241
a

astonishing-journalist-77684

12/01/2023, 10:16 PM
so, I ran the update but I am still getting the same error 😕
Copy code
Diagnostics:
  pulumi:pulumi:Stack (infra-dev-us-east-2):
    error: Error: invocation of aws:index/getPartition:getPartition returned an error: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
        at Object.callback (/Users/alex/git/rownd/infrastructure/node_modules/@pulumi/runtime/invoke.ts:159:33)
        at Object.onReceiveStatus (/Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/client.ts:338:26)
        at Object.onReceiveStatus (/Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
        at Object.onReceiveStatus (/Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
        at /Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24
        at processTicksAndRejections (node:internal/process/task_queues:77:11)
    error: Error: invocation of aws:index/getRegion:getRegion returned an error: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
        at Object.callback (/Users/alex/git/rownd/infrastructure/node_modules/@pulumi/runtime/invoke.ts:159:33)
        at Object.onReceiveStatus (/Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/client.ts:338:26)
        at Object.onReceiveStatus (/Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
        at Object.onReceiveStatus (/Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
        at /Users/alex/git/rownd/infrastructure/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24
        at processTicksAndRejections (node:internal/process/task_queues:77:11)

  aws:iam:Role (dev-us-east-2-eksRole-role):
    error: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
b

billowy-army-68599

12/01/2023, 10:17 PM
provider’s are stored in state as well, so you need to run a successful
pulumi up
after running
npm install --update
use https://github.com/jaxxstorm/aws-sso-creds to get some temporary credentials to update the stack provider
a

astonishing-journalist-77684

12/01/2023, 10:17 PM
how would I get to that now that it is broken in a way I can't work around?
b

billowy-army-68599

12/01/2023, 10:18 PM
or alternatively you can configure your aws config like this:
Copy code
[default]
region = [region]
output = json
[profile myprofile]
sso_account_id = [sso-account-id]
sso_role_name = [sso-role]
sso_region = [sso-region]
sso_start_url = [sso-start-url]
sso_registration_scopes = sso:account:access
region = [region]
output = json
and it’ll work
a

astonishing-journalist-77684

12/01/2023, 10:28 PM
let me try reverting my config to that
that configuration no longer works
b

billowy-army-68599

12/01/2023, 10:31 PM
oh? does
aws sts get-caller-identity
work?
a

astonishing-journalist-77684

12/01/2023, 10:34 PM
it does not
b

billowy-army-68599

12/01/2023, 10:34 PM
you likely need to remove
sso_registration_scopes
that’s a session specific property
a

astonishing-journalist-77684

12/01/2023, 10:34 PM
I did that. My config has the exact format and keys that you shared just now
b

billowy-army-68599

12/01/2023, 10:35 PM
mind sharing it?
a

astonishing-journalist-77684

12/01/2023, 10:35 PM
I have tried it with and without the # at the end of the start url
Copy code
[default]
region = [region]
output = json

[profile profile-name]
sso_account_id = [sso-acount-number]
sso_role_name = [sso-role-name]]
sso_region = [sso-region]
sso_start_url = [sso-start-url]
sso_registration_scopes = [sso-scope]
region = [region]
output = json
b

billowy-army-68599

12/01/2023, 10:38 PM
Copy code
sso_registration_scopes = [sso-scope]
you still have this property in there. Remove it, and reauth
a

astonishing-journalist-77684

12/01/2023, 10:38 PM
oh, the one you shared had it, so I thought I needed it
b

billowy-army-68599

12/01/2023, 10:39 PM
yes that was my mistake
a

astonishing-journalist-77684

12/01/2023, 10:40 PM
I removed all the cache files, removed the sso_registration_scopes, re-authorized through aws sso login --profile profile-name, then ran the sts get caller identity command and it is still broken
I have tried it with both the # at the end of the url and without it
ok, I left off
--profile
on the sts call, my mistake
sorry
b

billowy-army-68599

12/01/2023, 10:42 PM
there must be a misconfiguration in your profile, without being able to know all the required values, I can’t help much
ah, yes
I recommend using something like https://github.com/johnnyopao/awsp to switch profiles
a

astonishing-journalist-77684

12/01/2023, 10:42 PM
ah, there it goes. I got a working preview command 👍
b

billowy-army-68599

12/01/2023, 10:43 PM
the cache file problem will now likely been gone, as well
a

astonishing-journalist-77684

12/01/2023, 10:55 PM
I'll get this pushed shortly, I'm just writing everything up for the rest of my team. Thank you for your help, Lee. I really appreciate your time and patience helping me sort through this 🎉