https://pulumi.com logo
Join Slack
Powered by
Is ap-southeast-4 (Melbourne) available via Pulumi...
# aws
l
Is ap-southeast-4 (Melbourne) available via Pulumi? We've created a provider with 
region: "ap-southeast-4" as Region
, but as soon as it's used in a resource, we're getting:
Copy code
error: unable to validate AWS credentials.
    Details: no valid credential sources for  found.
    
    Please see
    for more information about providing credentials.
    
    Error: failed to refresh cached credentials, operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 36223ba0-c4c9-4a7c-b795-59636, api error InvalidClientTokenId: The security token included in the request is invalid
(Note the missing interpolated values in for found and _Please see _)
m
All regions are supported by Pulumi, but you might need to enable them first (an AWS thing independent of Pulumi): https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-regions.html
l
Yes, we've enabled Melbourne, all looks good in the console. If we switch the region of the specific provider to ap-southeast-1, it works as expected. When we switch it back.. it fails again 😞
I'm wondering if the fact that ap-southeast-4 isn't yet in the Region union type is causing difficulties. We've cast it to Region even though it's not in there, and it isn't complaining, but we haven't stepped into the code to see what's happening deep in the bowels...
m
Are you using the classic or the native provider?
l
Classic.
m
Ah, I see the problem. We missed switching it on on our end, negating my claim that Pulumi supports all regions…
We do but they have to be enabled once first
l
(I'm practicing my diagnoses 🙂 )
m
yep!
l
Cool. So I should park this work for a day or three? Then update pulumi-aws and try again?
m
If you can, yes. I’ll try to add it asap.
Just kicked off the release of v5.30.1 which will have the region.
l
Slightly better.. in that the interpolated values are now correct:
Copy code
error: unable to validate AWS credentials.
    Details: no valid credential sources for Pulumi AWS Classic found.
    
    Please see <https://www.pulumi.com/registry/packages/aws/installation-configuration/>
    for more information about providing credentials.
    
    Error: failed to refresh cached credentials, operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 8f032d1e-8e86-4dc0-af5a-41f980b, api error InvalidClientTokenId: The security token included in the request is invalid
    
    Make sure you have set your AWS region, e.g. `pulumi config set aws:region us-west-2`.
I don't know if it matters, but we're using MFA. I have just got a new session token, so that shouldn't be the problem? However, we're not assuming a role at any point (at least, not deliberately).
None of our source profiles specify a region. Region is provided only when constructing the aws.Provider.
I don't suppose anything is iterating over regions by name (rather than constant / enum) and might be getting fooled by the fact that there's no ap-southeast-3? Afaik it's the only gap in region names...
m
Hmm, you might be running into https://github.com/pulumi/pulumi-aws/issues/2344 which is high on our priorities. User reports are a little mixed but apparently there’s an issue with explicit providers and credentials. Sorry for the trouble!
l
Maybe.. but it applies only to ap-southeast-4 (that we've tried). We use explicit providers everywhere. Currently ap-southeast-1, ap-southeast-2 and us-west-2 are all fine.
I see this in a comment on that issue:
I have a theory that this is something to do with how default providers are resolved. Digging around the code for aws.Provider, it seems that if nothing is supplied for say region, it falls back to env variables. It probably first needs to fallback to the configured aws:region, and similarly for profile.
This is something we can test.
m
I’d be grateful for another data point. Although you’re right in that it shouldn’t make a difference what region you’re using,
l
Hmm.. apparently, region is always set:
Copy code
new aws.Provider(`${resourceContainer}${"ap-southeast-2" == region ? "" : "-" + region}`, {
        region: region,
        profile: stackToAccountInfoMap.get(resourceContainer)!.profileName
m
same with 
profile
, I assume? which contains all information necessary for authentication?
l
No regions in any profiles.
We use the same profile for connecting to any region, so thought it'd be wrong to specify a region.
m
True. I was referring to other auth-related properties like role_arn, sso_start_url etc.
l
We use source_profile + role_arn in he profile that is passed to the provider, and aws_access_key_id + aws_secret_access_key + aws_session_token (for MFA) in the source_profile.
(We'll switch to SSO soon, promise!)
m
That all sounds perfectly fine… I’ll keep digging, maybe not today though
l
Found it!
I just changed the the global endpoint validity to "Valid in all AWS regions".
image.png
So: new feature request, please use the regional STS endpoints!
I had to get a new session token, but once I did, it all worked.
m
Oh nice! Many thanks for figuring this out. Filed https://github.com/pulumi/pulumi-aws/issues/2406.
Open in Slack
PreviousNext
Powered by