https://pulumi.com logo
Title
d

dry-journalist-60579

03/13/2023, 10:12 PM
So I’m setting up Pulumi Deployments and have set up an OIDC provider and role in an AWS account that Pulumi deployment executor can assume. Now… any recommendations for set up if different stacks are in different AWS accounts? E.g. we have
dev
,
staging
, and
prod
stacks each in a separate account. Do I need to set up the OIDC provider and role to assume in every account? Or can I have one and somehow use cross-account role assuming?
l

little-cartoon-10569

03/14/2023, 1:14 AM
You can do either. I think which is better is dependent on how you access your accounts for other reasons. For example, if you prevent all direct access to sensitive accounts and require people to assume roles to get into those accounts, then you should probably stick with that approach for deployments. Having one role that allows deployments in your sensitive account is good, because you only have to make changes once in order to update deployments, whether it's from Pulumi Deployments (OIDC) or directly via a super-developer account of some sort. If you have no reason to use cross-account role assumption at the moment (e.g. SSO access to each account, managed by something like Okta), then an OIDC connection per account is a reasonable way to go. If neither of these conditions apply, then it probably comes down to personal preference. Do you prefer to have all your Pulumi-AWS auth configuration in a single role, with a single trust relationship? Easy to find all the info you ever need (on the topic) in one place, but also, easy to mess up typing all that JSON. For simpler resources spread out over more places (accounts), then have multiple OIDC connections.
d

dry-journalist-60579

03/14/2023, 1:34 AM
So I’m using Pulumi to bootstrap the creation of the OIDC provider and the role that the Pulumi Deployment executor will be assuming during automatic deployments to other stacks. This bootstrap stack is obviously one I need to run manually from the CLI locally. We basically have the following projects/stacks: • app-project ◦ dev-stack ◦ staging-stack ◦ prod-stack • bootstrap-project ◦ prod-stack And the following AWS accounts (set up à la Control Tower): • management (id:
XXXXXX1
) • workloads-app-dev (id:
XXXXXX2
) • workloads-app-staging (id:
XXXXXX3
) • workloads-app-prod (id:
XXXXXX4
) • workloads-cicd (id:
XXXXXX5
) In the bootstrap-project I am now creating a single OIDC provider and a role that Pulumi can assume using
AssumeRoleWithWebIdentity
in the
workloads-cicd
account. Now, I’ve set up the Pulumi Deployments configuration in the UI (anxiously awaiting https://github.com/pulumi/service-requests/issues/169 to not have to use the UI) to use the Role ARN created from the bootstrap stack. The app/dev stack targets the workloads-app-dev account, but I can’t quite figure out the right way to do this. I guess I’ll need a role within that account that the aforementioned role can assume and I’ll need to add that to my
aws:assumeRole:
in app-project/Pulumi.dev.yaml? Right now I’m getting an error during the deployment/update of:
AWS account ID not allowed: XXXXXX5
because I have a
aws:allowedAccountIds: [XXXXXX2]
in the app/dev stack to ensure the correct placement… Sorry if that is a lot of context. Ideally I’d prefer to not create the OIDC provider in each account.
l

little-cartoon-10569

03/14/2023, 1:43 AM
My suggestion would be to enhance the bootstrap-project code to cover the Control Tower work, including creating the roles in all the other accounts that trust the role in the workloads-cicd account. You might even create exports from that project for outputting Pulumi code that you can use as a template for creating AWS providers in projects that deploy to those accounts.
const stagingProvider = new aws.Provider("staging", { assumeRole: deploymentRoleInStagingAccount });
Or you might prefer to use profiles for that. That works too.
d

dry-journalist-60579

03/14/2023, 1:49 AM
Ah thank you, makes sense! I was leaning that way… so yeah, loop over all the accounts in my bootstrap stack to create a PulumiDeploymentRole that trusts the one in the
workloads-cicd
account. I can then specify that the app/dev stack should use the PulumiDeploymentRole either by explicitly creating a provider in the code or by specifying it in the Pulumi.dev.yaml:
config:
  aws:region: us-east-1
  aws:allowedAccountIds: [XXXXXX2]
  aws:assumeRole:
    roleArn: arn:aws:iam::XXXXXX2:role/PulumiDeploymentRole
l

little-cartoon-10569

03/14/2023, 1:51 AM
You can do that, though that relies on the default provider.. against which I have a personal gripe 🙂
And you definitely want to disallow default aws / awsx providers in the bootstrap project. That will definitely want many providers; if one is configured in the yaml file and all the others are configured in code, then ConfusionEnsues (tm).
d

dry-journalist-60579

03/14/2023, 1:53 AM
ahh interesting—do you always create an explicit provider?
l

little-cartoon-10569

03/14/2023, 1:53 AM
Always. Coding style guide / code reviews, Pulumi Policy, transformations and the Pulumi setting all prevent it.
I have wasted days and days recovering from problems caused by default providers.
d

dry-journalist-60579

03/14/2023, 1:54 AM
So interestingly I’ve found I have to use the default provider for my bootstrap stack because it’s the only way I can assume a role to assume a role as the
provider
code does not accept another provider
l

little-cartoon-10569

03/14/2023, 1:54 AM
I'd have to see that to understand...
d

dry-journalist-60579

03/14/2023, 1:54 AM
When running locally, I’m using aws sso
l

little-cartoon-10569

03/14/2023, 1:55 AM
Then use your SSO profile.
Explicitly, in code.
d

dry-journalist-60579

03/14/2023, 1:55 AM
so my active profile is:
role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_AWSAdministratorAccess_xxx
l

little-cartoon-10569

03/14/2023, 1:56 AM
You can use
new aws.Provider("bootstrap", { profile: new pulumi.Config("aws").profile })
.
Explicit provider, using your preferred profile.
d

dry-journalist-60579

03/14/2023, 1:57 AM
hmmmm 🧠 working….
hard to keep this all in my mental RAM
l

little-cartoon-10569

03/14/2023, 1:58 AM
The default AWS profile is just that, except with logic around AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY too.
If you set up your profile in Pulumi.bootstrap.yaml or whatever, and then use the default profile, another developer might accidentally leave those two env vars set, and they take priority over profile.
And that may cause stuff to get deleted from one account and created in another.. leading to
pulumi cancel
and clean-up duties.
d

dry-journalist-60579

03/14/2023, 2:00 AM
arn:aws:iam::XXX:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_AWSAdministratorAccess_xxx
=>
arn:aws:iam::XXX:role/service-role/AWSControlTowerAdmin
=>
arn:aws:iam::{0}:role/AWSControlTowerExecution
organization = aws.organizations.get_organization()

for account in organization.accounts:
    # Skip deactivated accounts and the management account
    if account.status != "ACTIVE":
        continue

    role_arn = pulumi.Output.format(
        "arn:aws:iam::{0}:role/AWSControlTowerExecution", account.id
    )

    # Create Provider to assume role
    provider = aws.Provider(
        f"Provider: {account.name}",
        assume_role={
            "roleArn": role_arn,
        },
    )

    password_policy = aws.iam.AccountPasswordPolicy(
        f"AccountPasswordPolicy: {account.name}",
        **PASSWORD_PARAMS,
        opts=pulumi.ResourceOptions(provider=provider),
    )
Basically I was trying to leverage the fact that control tower already sets up a
AWSControlTowerExecution
role in each account that can be assumed by the
service-role/AWSControlTowerAdmin
in the management account
meaning if I can assume that role, I’m able to loop over the accounts and assume
AWSControlTowerExecution
“out of the box”
l

little-cartoon-10569

03/14/2023, 2:04 AM
In the bootstrap project, that seems reasonable. I note that that account.id is an output but account.name isn't.. is that right? You might be missing some interpolation?
d

dry-journalist-60579

03/14/2023, 2:05 AM
hmm it seems to just work like that
this is the Resource that gets created
image.png
l

little-cartoon-10569

03/14/2023, 2:08 AM
Good stuff. The workloads provider is created before the loop? That looks right.
d

dry-journalist-60579

03/14/2023, 2:14 AM
Thank you for the help, @little-cartoon-10569!