https://pulumi.com logo
Powered by
Title
f

fresh-spring-82225

04/29/2023, 1:20 AM
I’m going to share a bug that won’t be reproducible 😢 but just in case anyone else runs across it …
I had some resources that were deleted externally to pulumi. When I tried to do a refresh in pulumi deployments, I got this a bunch of “not authorized” errors. Which is weird, since the stack config has 
aws:assumeRole
set to 
{"roleArn":"arn:aws:iam::ACCOUNT_ID:role/AWSControlTowerExecution"}
which means it should have permission to do anything 
aws:ec2:Instance (axial-instance): 
  error: refreshing urn:pulumi:dev::axial::aws:ec2/instance:Instance::axial-instance: 1 error occurred: 
  	* reading EC2 Instance (i-06d84518ff9f473e4): UnauthorizedOperation: You are not authorized to perform this operation. 
  	status code: 403, request id: a2712cf6-60af-48c8-a29f-0f678d60aa77 

aws:lambda:FunctionEventInvokeConfig (aca-invoke): 
  error: refreshing urn:pulumi:dev::axial::aws:lambda/functionEventInvokeConfig:FunctionEventInvokeConfig::aca-invoke: 1 error occurred: 
  	* reading Lambda Function Event Invoke Config (aca-function-f91f427): AccessDeniedException: 
  	status code: 403, request id: 1816cf65-e8e3-43e0-9761-5ee96162faa7 

aws:lambda:Function (aca-function): 
  error: refreshing urn:pulumi:dev::axial::aws:lambda/function:Function::aca-function: 1 error occurred: 
  	* reading Lambda Function (aca-function-f91f427): AccessDeniedException: 
  	status code: 403, request id: 297e078c-bffa-48df-b970-4a7374b33e39
I then ran 
pulumi refresh
from the command line and it completed successfully, correctly noting that the resources had been deleted: 
-   ├─ aws:lambda:FunctionEventInvokeConfig  aca-invoke               deleted (1s)     
     ├─ command:local:Command                 axial-get-date                            
     ├─ pulumi:pulumi:StackReference          monolith-corp/trc/dev                     
 -   ├─ aws:lambda:Function                   aca-function             deleted (1s)     
 -   └─ aws:ec2:Instance                      axial-instance           deleted (1s)
So, obviously I can’t reproduce the bug now 😕 but as I said, just in case anyone else runs into this situation …
working on this further, I think the problem is that my 
aws:assumeRole
was not respected
l

lemon-agent-27707

04/30/2023, 12:55 PM
Are you using OIDC?
f

fresh-spring-82225

05/01/2023, 5:07 AM
Yes
l

lemon-agent-27707

05/01/2023, 2:03 PM
If you think there is a bug here, would love to get an issue open: github.com/pulumi/pulumi-cloud-requests/issues/new Are you able to configure OIDC to use that role directly instead of assuming into it once you get OIDC credentials?
From a little searching, it sounds like assume role with OIDC might require you to set up some extra IAM roles. There is a package you could try to use (haven't tried it myself): https://www.pulumi.com/registry/packages/aws-iam/api-docs/assumablerolewithoidc/ Or the underlying reference implementation might be useful just to look at: https://github.com/pulumi/pulumi-aws-iam/blob/master/provider/pkg/provider/assumable_role_with_oidc.go#L146-L152
@dry-journalist-60579 I believe you were using assume role with OIDC. Did you end up getting it working? Curious if you ever tried out the resources linked above.
d

dry-journalist-60579

05/01/2023, 3:05 PM
Oh yeah we have been using OIDC smoothly with pulumi deployments
We have a stack that bootstraps the creation of that role and maintains it going forward. We then specify that role in the assume role config for all our deployments
f

fresh-spring-82225

05/01/2023, 4:14 PM
Here’s what I’ve done so far: • created an identity provider in my control tower management account, arn:aws:iam:MGT_ACCT_ID:oidc-provider/api.pulumi.com/oidc • created a role for pulumi deployments to assume, arn:aws:iam:MGT_ACCT_ID:role/PulumiDeploymentsRole ◦ trust relationship allows the above federated principal to assume the role ◦ permission policy is the AWSControlTowerStackSetRolePolicy created by control tower, which allows assuming arn:aws:iam::*:role/AWSControlTowerExecution • in my application stack, I have 
aws:assumeRole
set as follows: 
aws:assumeRole:
    roleArn: arn:aws:iam::APP_ACCT_ID:role/AWSControlTowerExecution
The error messages I’m now seeing on running update in pulumi deployments: 
command:local:Command [...] creating (12s) An error occurred (AccessDeniedException) when calling the GetAuthorizationToken operation: User: arn:aws:sts:MGT_ACCT_ID:assumed-role/PulumiDeploymentsRole/pulumi is not authorized to perform: ecr:GetAuthorizationToken on resource: * because no identity-based policy allows the ecr:GetAuthorizationToken action
which tells me the 
assumeRole
setting in my 
Pulumi.dev.yaml
file isn’t taking effect
d

dry-journalist-60579

05/01/2023, 4:20 PM
@fresh-spring-82225 for what it’s worth, here is the Pulumi code I used to create the OIDC role that works for us: 
import pulumi
import pulumi_aws as aws
import json

# Create OIDC provider for Pulumi Deployments
oidc_provider = aws.iam.OpenIdConnectProvider(
    "Pulumi OIDC Provider",
    client_id_lists=["myorg"],
    thumbprint_lists=["9e99a48a9960b14926bb7f3b02e22da2b0ab7280"],
    url="<https://api.pulumi.com/oidc>",
)

oidc_provider_role = aws.iam.Role(
    "Pulumi OIDC Provider Role",
    name="PulumiOIDC",
    assume_role_policy=oidc_provider.arn.apply(
        lambda arn: json.dumps(
            {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Effect": "Allow",
                        "Principal": {
                            "Federated": arn,
                        },
                        "Action": "sts:AssumeRoleWithWebIdentity",
                        "Condition": {
                            "StringEquals": {
                                "<http://api.pulumi.com/oidc:aud|api.pulumi.com/oidc:aud>": "myorg",
                            }
                        },
                    }
                ],
            }
        )
    ),
    managed_policy_arns=["arn:aws:iam::aws:policy/AdministratorAccess"],
)

pulumi.export("PulumiOIDCRoleArn", oidc_provider_role.arn)
f

fresh-spring-82225

05/01/2023, 6:42 PM
@dry-journalist-60579 I recreated this (in typescript) for my management account, and it seems to have unblocked my application stack. The difference is that I was using AWSControlTowerStackSetRolePolicy instead of AdministratorAccess
d

dry-journalist-60579

05/01/2023, 6:46 PM
Ah nice! I’m not sure what the proper least privilege configuration is but at least it’s working for now :)
2 Views
#pulumi-deploymentsJoin Slack
Powered by