https://pulumi.com logo
Powered by
Title
l

little-cartoon-10569

03/07/2023, 10:04 PM
Is ap-southeast-4 (Melbourne) available via Pulumi? We've created a provider with 
region: "ap-southeast-4" as Region
, but as soon as it's used in a resource, we're getting: 
error: unable to validate AWS credentials.
    Details: no valid credential sources for  found.
    
    Please see
    for more information about providing credentials.
    
    Error: failed to refresh cached credentials, operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 36223ba0-c4c9-4a7c-b795-59636, api error InvalidClientTokenId: The security token included in the request is invalid
(Note the missing interpolated values in for found and _Please see _)
m

melodic-tomato-39005

03/07/2023, 10:10 PM
All regions are supported by Pulumi, but you might need to enable them first (an AWS thing independent of Pulumi): https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-regions.html
l

little-cartoon-10569

03/07/2023, 10:11 PM
Yes, we've enabled Melbourne, all looks good in the console. If we switch the region of the specific provider to ap-southeast-1, it works as expected. When we switch it back.. it fails again 😞
I'm wondering if the fact that ap-southeast-4 isn't yet in the Region union type is causing difficulties. We've cast it to Region even though it's not in there, and it isn't complaining, but we haven't stepped into the code to see what's happening deep in the bowels...
m

melodic-tomato-39005

03/07/2023, 10:14 PM
Are you using the classic or the native provider?
l

little-cartoon-10569

03/07/2023, 10:15 PM
Classic.
m

melodic-tomato-39005

03/07/2023, 10:20 PM
Ah, I see the problem. We missed switching it on on our end, negating my claim that Pulumi supports all regions…
We do but they have to be enabled once first
l

little-cartoon-10569

03/07/2023, 10:20 PM
Is that anything to do with this code? https://github.com/pulumi/pulumi-aws/blob/93273a52a25c4a33795a0c71ebd67ed2ec87740c/provider/resources.go#L3723
(I'm practicing my diagnoses 🙂 )
m

melodic-tomato-39005

03/07/2023, 10:21 PM
yep!
l

little-cartoon-10569

03/07/2023, 10:21 PM
Cool. So I should park this work for a day or three? Then update pulumi-aws and try again?
m

melodic-tomato-39005

03/07/2023, 10:21 PM
If you can, yes. I’ll try to add it asap.
Just kicked off the release of v5.30.1 which will have the region.
l

little-cartoon-10569

03/08/2023, 5:20 AM
Slightly better.. in that the interpolated values are now correct: 
error: unable to validate AWS credentials.
    Details: no valid credential sources for Pulumi AWS Classic found.
    
    Please see <https://www.pulumi.com/registry/packages/aws/installation-configuration/>
    for more information about providing credentials.
    
    Error: failed to refresh cached credentials, operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 8f032d1e-8e86-4dc0-af5a-41f980b, api error InvalidClientTokenId: The security token included in the request is invalid
    
    Make sure you have set your AWS region, e.g. `pulumi config set aws:region us-west-2`.
I don't know if it matters, but we're using MFA. I have just got a new session token, so that shouldn't be the problem? However, we're not assuming a role at any point (at least, not deliberately).
None of our source profiles specify a region. Region is provided only when constructing the aws.Provider.
I don't suppose anything is iterating over regions by name (rather than constant / enum) and might be getting fooled by the fact that there's no ap-southeast-3? Afaik it's the only gap in region names...
m

melodic-tomato-39005

03/08/2023, 10:32 PM
Hmm, you might be running into https://github.com/pulumi/pulumi-aws/issues/2344 which is high on our priorities. User reports are a little mixed but apparently there’s an issue with explicit providers and credentials. Sorry for the trouble!
l

little-cartoon-10569

03/08/2023, 10:45 PM
Maybe.. but it applies only to ap-southeast-4 (that we've tried). We use explicit providers everywhere. Currently ap-southeast-1, ap-southeast-2 and us-west-2 are all fine.
I see this in a comment on that issue:
I have a theory that this is something to do with how default providers are resolved. Digging around the code for aws.Provider, it seems that if nothing is supplied for say region, it falls back to env variables. It probably first needs to fallback to the configured aws:region, and similarly for profile.
This is something we can test.
m

melodic-tomato-39005

03/08/2023, 10:48 PM
I’d be grateful for another data point. Although you’re right in that it shouldn’t make a difference what region you’re using,
l

little-cartoon-10569

03/08/2023, 10:49 PM
Hmm.. apparently, region is always set: 
new aws.Provider(`${resourceContainer}${"ap-southeast-2" == region ? "" : "-" + region}`, {
        region: region,
        profile: stackToAccountInfoMap.get(resourceContainer)!.profileName
m

melodic-tomato-39005

03/08/2023, 10:54 PM
same with 
profile
, I assume? which contains all information necessary for authentication?
l

little-cartoon-10569

03/08/2023, 10:54 PM
No regions in any profiles.
We use the same profile for connecting to any region, so thought it'd be wrong to specify a region.
m

melodic-tomato-39005

03/08/2023, 10:55 PM
True. I was referring to other auth-related properties like role_arn, sso_start_url etc.
l

little-cartoon-10569

03/08/2023, 10:57 PM
We use source_profile + role_arn in he profile that is passed to the provider, and aws_access_key_id + aws_secret_access_key + aws_session_token (for MFA) in the source_profile.
(We'll switch to SSO soon, promise!)
m

melodic-tomato-39005

03/08/2023, 10:58 PM
That all sounds perfectly fine… I’ll keep digging, maybe not today though
l

little-cartoon-10569

03/08/2023, 11:05 PM
Found it!
I actually asked about it a few days ago: https://pulumi-community.slack.com/archives/CRH5ENVDX/p1678158942826929
I just changed the the global endpoint validity to "Valid in all AWS regions".
image.png
So: new feature request, please use the regional STS endpoints!
I had to get a new session token, but once I did, it all worked.
m

melodic-tomato-39005

03/08/2023, 11:16 PM
Oh nice! Many thanks for figuring this out. Filed https://github.com/pulumi/pulumi-aws/issues/2406.
#awsJoin Slack
Powered by