https://pulumi.com
Join Slack
Hi everyone. I've been banging my head against a wall for hours trying to figure out how to use AWS ...
# getting-started
b
Hi everyone. I've been banging my head against a wall for hours trying to figure out how to use AWS OIDC in a brand new AWS account configured with Control Tower. I have a management account and I'm trying to deploy into a sub-account. I created the OIDC role in the management account and created a role in the sub-account that it can assume with PowerUser access. I have my config file set up to use ESC dynamic variables. No matter what I do I get:
Copy code
Details: validating provider credentials: retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: e6482c74-b294-4f9d-80c3-2f217408d02c, api error ExpiredToken: The security token included in the request is expired
However, when I run aws sts assume-role to the sub-account role from my management account, it works fine. Not sure how to debug this farther.
l
I remember this happening to me. But I have the memory of a goldfish, so the solution eludes me. I'll see if I can find any notes I made about this or example code about how to get it working.
b
Claude is sending me in circles trying to debug it.
l
How are you using the OIDC connection? Are you constructing a Provider in code? Using the default provider? I seem to recall that you need a web identity role rather than a simple role, does that sound right?
Ah no, you can use a simple role, I see that in my code.
I note that I have to omit the region when I construct a Provider in the case where OIDC is in use.
b
Oh interesting
Do you mind sharing what your provider setup looks like and also what values you put in the local .yml file?
l
The stack config file contains nothing: in fact it contains the aws:region, but in my Provider-constructor code, I deliberately don't use that value when I detect OIDC is in use.
My Provider constructor when OIDC is in use just sets the 
assumeRole
arg, and nothing else. The role is the the one I created via IAM Identity Centre, with the OIDC checks added in there
b
It's funny there's example code on github for how to create the OIDC stuff but not how to use it (that I can find). https://github.com/pulumi/examples/blob/master/aws-ts-oidc-provider-pulumi-cloud/index.ts
l
Well you don't generally use it: it's only used from something like Pulumi ESC or Pulumi Ddeployments. It doesn't work well for interactive use.
So long as you're not setting any creds directly, the Provider just works with OIDC.
Just set the assumeRole or assumeRoleWithWebIdentity args, and roll with it.
I presume that the OIDC environment has the session token set, and that's all that's needed.
I never really looked into the magic of it.
b
Interesting, ok.
so when you do 
pulumi up
it's doing it entirely through the OIDC stuff? Or is that using a local aws profile?
l
Pulumi code that's intended to work with OIDC should (probably) also support normal interactive methods of use like SSO or AWS_ACCESS_KEY_ID.
Pulumi itself does not do anything with OIDC. That has to be done in the environment that invokes Pulumi.
Which is why you just assume a role or whatever.
Pulumi ESC uses OIDC to get the session token and invoke Pulumi; same for Pulumi Deployments.
So by the time Pulumi is running, the environment it's running in already has the OIDC stuff in it.
I had to create this little treasure to detect if my program was running in an OIDC environment. I bet there's a better way, but this is what's working for me:
Copy code
function awsOidcIsCurrent(): boolean {
  const stdout = execFileSync("aws", ["configure", "list"]).toString("ascii");
  const lines = stdout.split(EOL);
  const accessKeyLine = lines.find((line) => line.includes("access_key"));
  if (accessKeyLine === undefined) {
    return false;
  }
  return accessKeyLine.includes("env");
}
Try it from the command line: run 
aws configure list
and see if the access_key includes 
env
I had deleted all my ESC environments! Anyway, I got it up again, and this is what I get with a working OIDC connection:
Copy code
$ pulumi env run myproj/myenv aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************YJV3              env    
secret_key     ****************rxeZ              env    
    region                <not set>             None    None
So the Type is env, and as far as I can tell, that happens only when I run via ESC.
b
Copy code
pulumi env run kickplan/oidc-test aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key                <not set>             None    None
secret_key                <not set>             None    None
    region                us-west-2      config-file    ~/.aws/config
Ah ha!
That helps!
l
Yea you won't have much success with those details. Your ESC config must be wrong?
b
Copy code
pulumi env run kickplan/oidc-test aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************I37Z              env
secret_key     ****************0OE8              env
    region                us-west-2      config-file    ~/.aws/config
Yeah, that helped so much
My trust policy was wrong.
l
Aha! Nice one.
b
Copy code
// Create an AWS provider
const provider = new aws.Provider("default", {
	region: 'us-west-2'
});
How do you instantiate the provider? This always gives me 
pulumi:providers:aws resource 'default' has a problem: No valid credential sources found.
l
With OIDC creds, you must assume a role. Use either 
assumeRole
or 
assumeRoleWithIdentity
args in the constructor.
b
Hey I got it!
We did it!
Thanks for your help!
s
Hello. I am getting a similar error and it is driving me crazy. I have a pulumi environment
Copy code
values:
  aws:
    login:
      fn::open::aws-login:
        oidc:
          duration: 1h
          roleArn: arn:aws:iam::{roleArn}
          sessionName: pulumi-environments-session
          subjectAttributes:
            - currentEnvironment.name
            - pulumi.user.login
  pulumiConfig:
    aws:accessKey: ${aws.login.accessKeyId}
    aws:secretKey: ${aws.login.secretAccessKey}
    aws:token: ${aws.login.sessionToken}
    aws:region: us-east-1
The AWS oidc role's trust relationships:
Copy code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::243709619272:oidc-provider/api.pulumi.com/oidc"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "<http://api.pulumi.com/oidc:aud|api.pulumi.com/oidc:aud>": "{mypulumiorg}"
        }
      }
    }
  ]
}
the stack I am trying to update has a very simple yml with just the 
environment
key When running 
pulumi up
or 
refresh
I get a bunch of
Copy code
error: Preview failed: 1 error occurred:
    	* Retrieving AWS account details: validating provider credentials: retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 0ac7335b-9092-4a96-9a63-a8274e7d47f8, api error ExpiredToken: The security token included in the request is expired
Copy code
pulumi env run my/env/path aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key                <not set>             None    None
secret_key                <not set>             None    None
    region                <not set>             None    None
@boundless-ice-8460 @little-cartoon-10569 any idea of what it is happening?
The problem happens only when deleting resources. If I add a new one everything works fine. Moreover, in the environment I've set also environment variables and they are correctly retrieved
l
If you do both in the same 
pulumi up
does it still happen? If not, that suggests that something is not requesting a new temporary token when the only interactions with the provider are deletes. Which sounds very much like a bug.
s
What do you mean by "If you do both in the same 
pulumi up
does it still happen?"
if you mean deleting and adding: in such a case the 
pulumi up
command in this case creates the resources needed but fail in deleting
10 Views
Open in Slack
PreviousNext
Powered by

Pulumi Community

No matter how you like to participate in developer communities, Pulumi wants to meet you there. If you want to meet other Pulumi users to share use-cases and best practices, contribute code or documentation, see us at an event, or just tell a story about something cool you did with Pulumi, you are part of our community.

Powered by