dry-journalist-60579
03/16/2023, 10:40 PMrole/aws-reserved/sso.amazonaws.com/AWSReservedSSO_AWSAdministratorAccess_xxxxxxxxx (the role I am logged into via the CLI)
=> assumes
role/service-role/AWSControlTowerAdmin (via Pulumi.<stack>.yaml config var aws:assumeRole.roleArn)
=> assumes
role/AWSControlTowerExecution (via `aws.Provider` in the Pulumi code)
locally this works. When running via Pulumi Deployments it doesn’t seem to. The Deployment “configuration” seems to show the correct aws:assumeRole
value but it doesn’t get assumed. It basically skips the middle assumption and uses the Deployment role to try to directly assume the 3rd role, AWSControlTowerExecution and it fails.
If anything I’d expect to see that assumed-role/PulumiOIDC/pulumi
cannot assume role/service-role/AWSControlTowerAdmin
or that role/service-role/AWSControlTowerAdmin
cannot assume role/AWSControlTowerExecution
, but I’m getting assumed-role/PulumiOIDC/pulumi
cannot assume role/AWSControlTowerExecution
.
This makes me think that the deploy executor is not respecting the Pulumi.<stack>.yaml
configuration…AWSControlTowerAdmin
can assume AWSControlTowerExecution
in every account and it already has the AdministratorAccess
policy attached. This means that I don’t have to update/create roles/policies in new accounts when they get created. This particular Pulumi stack loops over all the accounts in the organization to set up some standard controls (like a password policy).
For developer ease of use, I didn’t want people to have to assume the control tower admin role before running the stack. Rather, I just wanted people to use their SSO roles. That’s why I wanted to use AWSReservedSSO_AWSAdministratorAccess_xxxxxxxxx
to assume role/service-role/AWSControlTowerAdmin
in the management account to assume role/AWSControlTowerExecution
in each account in the loop. The only way to do this was to leverage the Pulumi config and the default provider. Unfortunately there’s no way to do it in the code as you cannot use a Provider to create a Provider.Pulumi.yaml
setting like aws:assumeRole.roleArn
something that is supposed to be used by the Pulumi operation during Pulumi Deployments?red-match-15116
03/17/2023, 5:42 PMdry-journalist-60579
03/17/2023, 6:02 PMlemon-agent-27707
03/17/2023, 6:58 PMWith this configuration, each deployment of this stack will attempt to exchange the deployment’s OIDC token for AWS credentials using the specified IAM role prior to running any pre-commands or Pulumi operations. The fetched credentials are published in theMight be helpful to share code on how you are configuring the provider and assuming the roles.,AWS_ACCESS_KEY_ID
, andAWS_SECRET_ACCESS_KEY
environment variables. The raw OIDC token is also available for advanced scenarios in theAWS_SESSION_TOKEN
environment variable and thePULUMI_OIDC_TOKEN
file./mnt/pulumi/pulumi.oidc
dry-journalist-60579
03/17/2023, 7:21 PMconfig:
aws:region: us-east-1
aws:allowedAccountIds: [xxx]
aws:assumeRole:
roleArn: arn:aws:iam::xxx:role/service-role/AWSControlTowerAdmin
This is my `Pulumi.yaml`:
name: admin
runtime:
name: python
description: Manage resources related to our AWS account structure
import pulumi
import pulumi_aws as aws
MANAGEMENT_ACCOUNT_ID = "xxx"
PASSWORD_PARAMS = {
"require_uppercase_characters": True,
"require_lowercase_characters": True,
"require_symbols": True,
"require_numbers": True,
"minimum_password_length": 20,
}
organization = aws.organizations.get_organization()
for account in organization.accounts:
# Skip deactivated accounts and the management account
if account.status != "ACTIVE":
continue
if account.id == MANAGEMENT_ACCOUNT_ID:
# See aws-admin-bootstrap-stack where the ManagementAccountAdmin role is created
provider = aws.Provider(
f"Provider: Mighty Management Account",
region="us-east-1",
assume_role={
"roleArn": "arn:aws:iam::xxx:role/ManagementAccountAdmin",
},
)
else:
role_arn = pulumi.Output.format(
"arn:aws:iam::{0}:role/AWSControlTowerExecution", account.id
)
# Create Provider to assume role
provider = aws.Provider(
f"Provider: {account.name}",
region="us-east-1",
assume_role={
"roleArn": role_arn,
},
)
password_policy = aws.iam.AccountPasswordPolicy(
f"AccountPasswordPolicy: {account.name}",
**PASSWORD_PARAMS,
opts=pulumi.ResourceOptions(provider=provider),
)
aws-sso-util
and awsume
locally… when I “awsume” the SSO role before I run Pulumi up, it does set AWS_*
env varsenv | grep AWS
AWSUME_COMMAND=Mighty-Management-Account.AWSAdministratorAccess
AWS_ACCESS_KEY_ID=XXX
AWS_SECRET_ACCESS_KEY=YYY
AWS_SESSION_TOKEN=ZZZ
AWS_REGION=us-east-1
AWS_DEFAULT_REGION=us-east-1
AWSUME_PROFILE=Mighty-Management-Account.AWSAdministratorAccess
AWSUME_EXPIRATION=2023-03-17T20:23:39
poetry run pulumi up
and it runs correctlylemon-agent-27707
03/19/2023, 7:28 PMIt basically skips the middle assumption and uses the Deployment role to try to directly assume the 3rd role, AWSControlTowerExecution and it fails.I don't see anywhere in your code where there is an explicit attempt to assume
role/service-role/AWSControlTowerAdmin
Is it possible that you are using an account/role locally that has broader permissions than the role you've assigned to the OIDC identity provider?
Another way of saying this, is it possible that locally you are directly assuming AWSControlTowerExecution
and that your starting role happens to have appropriate permissions to do this? It looks like from your env, that you are using the administrator access profile locally, so it would not surprise me if this has greater permissions than the role you assigned to the OIDC provider.AWSControlTowerAdmin
and then specify that provider when creating the second provider that assumes AWSControlTowerExecution
dry-journalist-60579
03/20/2023, 2:54 PMrole/service-role/AWSControlTowerAdmin
role gets assumed is via the Pulumi.prod.yaml:TypeError: Explicit providers may not be used with provider resources
role/AWSControlTowerExecution
is actually the root for the management account meaning that I don’t have to assume an intermediary role…lemon-agent-27707
03/21/2023, 12:08 AMdry-journalist-60579
03/21/2023, 5:47 PMlemon-agent-27707
03/21/2023, 10:34 PMI think the issue here is that the project/stack configuration from the Pulumi.yaml files are being ignored during the Pulumi Deployments…?Pulumi Deployments still use the CLI when running an update. My guess is that there are a different set of environment variables that get set during a deployment vs what you have set locally. It is possible that the different env vars are causing some difference in behavior between the precedence of explicit configuration, stack configuration, and environment variables as is mentioned in the issue.
dry-journalist-60579
03/21/2023, 10:51 PMAWSControlTowerExecution
role within the organization’s sub-accounts seamlessly (both via explicit provider created in code and via config.aws:assumeRole.roleArn
specified in Pulumi.yaml files)! 😛arty-pizza:lemon-agent-27707
03/22/2023, 8:24 PMbrash-gigabyte-81569
04/18/2023, 2:55 PMerror: could not validate provider configuration: 1 error occurred:
* Invalid or unknown key
when i provide something like so
config:
aws:region: us-east-1
aws:assumeRole.roleArn: *************
dry-journalist-60579
04/18/2023, 3:05 PMconfig:
aws:assumeRole:
roleArn: xxx
aws:region: us-east-1
brash-gigabyte-81569
04/18/2023, 3:07 PMunexpected configuration type 'map[string]interface {}': valid types are string, List<string>, number, List<number>, integer, List<integer>, boolean, List<number>
dry-journalist-60579
04/18/2023, 3:09 PMbrash-gigabyte-81569
04/18/2023, 3:10 PMdry-journalist-60579
04/18/2023, 3:15 PMbrash-gigabyte-81569
04/18/2023, 3:15 PMdry-journalist-60579
04/18/2023, 3:16 PMbrash-gigabyte-81569
04/18/2023, 3:17 PMdry-journalist-60579
04/18/2023, 3:17 PMyaml
for the whole shebang rather than TS/python/etc.?brash-gigabyte-81569
04/18/2023, 3:18 PMdry-journalist-60579
04/18/2023, 3:20 PMbrash-gigabyte-81569
04/18/2023, 3:21 PMlemon-agent-27707
04/24/2023, 2:25 AMbrash-gigabyte-81569
04/24/2023, 1:06 PM