https://pulumi.com logo
#aws
Title
# aws
r

rhythmic-branch-12845

07/29/2022, 8:52 AM
can Pulumi be run from an EC2 instance that has an instance profile assigned to it? or must we set up AWS credentials?
s

strong-helmet-83704

07/29/2022, 3:46 PM
you should be able to run it from any system which is api authenticated.
l

little-cartoon-10569

07/31/2022, 9:51 PM
You should consider using a role that you assume, but you don't need extra credentials. The instance profile is exposed within the instance via the access and secret key env vars, and Pulumi will use those if no other creds (e.g. default profile) are set up.
r

rhythmic-branch-12845

08/01/2022, 3:42 AM
I am expecting to use an instance profile, which basically assigns a role to the EC2 instance itself. This is not exposed as your standard env vars, though (that is the point, to prevent having to deal with env vars). My understanding is that anything that uses the AWS SDK will be able to work with this setup, but it looks like Pulumi does not use the SDK? This what I get in the EC2 instance, without any “AWS_*” env vars:
Copy code
Diagnostics:
  aws:ec2:Instance (api):
    error: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
l

little-cartoon-10569

08/01/2022, 4:45 AM
As I understand it, the instance profile is available through the EC2 metadata, and it's the lowest priority source of creds. I'm not certain that Pulumi's AWS provider uses it. Therefore, when I run from EC2, I've always pulled the instance profile values out to ACCESS and SECRET env vars, and it's worked without a hitch.
You can grab the values by wgetting / curling this URL, and piping through jq or similar: http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance
r

rhythmic-branch-12845

08/01/2022, 4:54 AM
thank you. That’s helpful to know! I’m expecting Pulumi to do better, but I guess until it does, this is how I will have to get this working. Thank you!
this was working for a while.. but I have 2 issues right now: 1. this first one is the “obvious” one: it looks like to survive with using the instance profile BUT without having the underlying logic of the SDK to help me out, I would basically have to duplicate this aws credential pulling method myself every single time I call
pulumi up
. That’s because these credentials aren’t permanent, and have a timeout 2. leaving number 1 aside, it was working for a while for me… but after changing the permissions of the IAM role (it currently allows for
"*", "*"
), I am getting
AuthFailure: AWS was not able to validate the provided access credentials
and Pulumi no longer works?
it appears that there might be some bug with using temporary creds or something. No matter what I did, i was not able to get this resolved. I had to switch to using non-temporary creds, and then everything was ok 😠
l

little-cartoon-10569

08/01/2022, 9:09 PM
Did you set the session token too? You need 3 env vars for temp creds.
You don't need to duplicate the cred-pulling method if you use an assumed role for Pulumi, which is what I recommend. You can create a profile that assumes role X and uses the default profile as its source (configure this in ~/.aws/config). Since the default profile uses the instance profile, you don't need to do anything else: so long as role X can be assumed by the instance profile role, it works every time.
It's also a lot safer: if an attacker gains access to your EC2 instance and the instance profile gives them Pulumi-like powers, they can destroy your infrastructure. If you use the role-assumption requirement, they don't have those powers. They need to know that they have to assume the extra role before they get those powers.
r

rhythmic-branch-12845

08/02/2022, 11:17 AM
I did set the session token, yes. It was working all along until somehow it didn’t anymore. Sorry, but I’m somewhat confused. When I create my instance profile, I already tie it to the role that I want, and this role has the required permissions. Could you give an example for ~/.aws/config? is there anything I need to do with or within Pulumi to assume this role?
I’m trying the following in ~/.aws/config (and nothing in credentials):
Copy code
[default]
role_arn = arn:aws:iam::<ACCOUNT_ID>:role/pulumi
And then
pulumi config set --path aws:assumeRole.roleArn arn:aws:iam::<ACCOUNT_ID>:role/pulumi
. No luck:
error: Preview failed: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
l

little-cartoon-10569

08/02/2022, 8:34 PM
While you can do all this in Pulumi, I suggest doing the role assumption in the AWS config file. Role assumption in config files is introduced here: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html. Look for uses of source_profile. In your config file, you can set up a profile that uses the default profile (= instance profile) as its source_profile. Your Pulumi provider can refer to the name of that profile. There isn't any particular advantage to assuming the role in the config file other than convenience: when I run my code locally, I have a simple profile with admin privs for my dev AWS account, and I don't need to do anything special in my Pulumi code to detect whether it needs to assume a role (in prod) or not (in dev). So long as my simple profile has the same name as the nice assumed-role profile, Pulumi can't tell the difference and it just works.
r

rhythmic-branch-12845

08/03/2022, 3:21 AM
it sounds like what you’re saying is to do this? ~/.aws/config:
Copy code
[pulumi]
role_arn = arn:aws:iam::<ACCOUNT_ID>:role/pulumi
source_profile = default
(no need for ~/.aws/credentials) Pulumi.dev.yaml:
Copy code
config:
  aws:region: ap-southeast-1
  aws:profile: pulumi
With this setup, I still get the dreaded
error: Preview failed: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
Is my setup correct?
I’ve also tried this in ~/.aws/config, with the same error:
Copy code
[pulumi]
role_arn = arn:aws:iam::<ACCOUNT_ID>:role/pulumi
credential_source = Ec2InstanceMetadata
l

limited-rainbow-51650

08/03/2022, 6:36 AM
@rhythmic-branch-12845 if you have an EC2 instance with an instance profile attached, you only have to specify the AWS region via an env variable or stack config. The AWS SDK for Go is integrated in both the AWS Classic (bridged from the TF AWS provider) and our new AWS Native provider. This SDK, like any other AWS SDK, implements the standard credential lookup like this: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials With an instance profile attached, the credentials are not exposed as env variables, but these are retrieved by the SDK on the EC2 metadata service, which is the URL posted by @little-cartoon-10569 as a previous answer in this thread. You don’t have to do this manually.
r

rhythmic-branch-12845

08/03/2022, 6:38 AM
right. This is what I’m expecting: “config-free setup”. But it doesn’t seem to work as I expect. And given the error messages and the pointing to https://pulumi.io/install/aws.html where it’s all about setting up credentials, it’s made me wonder. NOTE: I am not using Go, but Python
l

limited-rainbow-51650

08/03/2022, 6:42 AM
This info is more aimed towards your local workstation. I’ll make sure that the EC2 instance profile variant get’s added to that. Now, unrelated to the programming language you choose to implement your infrastructure code, our providers are implemented in Go, so that is why I referred to the AWS SDK for Go. It is the AWS provider binary which executes the AWS authentication.
r

rhythmic-branch-12845

08/03/2022, 6:43 AM
ok gotcha. Thank you for the clarification. If this is the case then I am truly stuck. Because I already have the instance profile attached to my EC2 instance. And until I use credentials, I am not able to get Pulumi working
l

limited-rainbow-51650

08/03/2022, 6:44 AM
@rhythmic-branch-12845 ok, let me dig further on this for you.
🙏 1
👍 1
@rhythmic-branch-12845 via the Github issues of our aws provider, I bumped into this link: https://github.com/pulumi/pulumi-aws#authenticating-pulumi-aws-via-ec2-instance-metadata So I think you will have to re-enable the use of EC2 metadata by setting this in your stack configuration:
Copy code
pulumi config set aws:skipMetadataApiCheck false
r

rhythmic-branch-12845

08/03/2022, 6:53 AM
this looks promising, and maybe there is something to it. I’m not there yet. So this is my Pulumi.dev.yaml:
Copy code
config:
  aws:region: ap-southeast-1
  aws:skipMetadataApiCheck: "false"
And I’ve got absolutely nothing in ~/.aws now. And I still get
error: Preview failed: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
l

limited-rainbow-51650

08/03/2022, 6:57 AM
I assume you are logged in on that EC2 instance at the moment? If so, can you try this:
curl <http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance>
Do you get valid credentials back in the returned JSON document?
r

rhythmic-branch-12845

08/03/2022, 7:01 AM
yes, I am logged in. The curl works and returns credentials.
l

limited-rainbow-51650

08/03/2022, 8:10 AM
I don’t know what permissions the EC2 instance have via the instance profile and attached role(s). Can you run a simple AWS cli command to test?
aws ec2 describe-instances --instance-ids <your-instance-id>
r

rhythmic-branch-12845

08/03/2022, 10:26 AM
not sure if this’ll give you what you’re looking for, but I’ll DM you
l

limited-rainbow-51650

08/03/2022, 12:30 PM
@rhythmic-branch-12845 I wasn’t really interested in the output, but the AWS CLI uses the same credential lookup strategy as the provider, given it is based on the same AWS SDK. If running that command with the AWS CLI works, it should work for Pulumi as well.
Can you perform these troubleshooting actions on your EC2 instance? https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_iam-ec2.html
@rhythmic-branch-12845 also, can you try to set this on your stack config:
pulumi config set aws:skipCredentialsValidation true
r

rhythmic-branch-12845

08/03/2022, 1:42 PM
re https://pulumi-community.slack.com/archives/CRH5ENVDX/p1659529800534399?thread_ts=1659084774.882929&amp;cid=CRH5ENVDX: sorry, I thought you wanted the output! this wasn’t run on the instance itself (no AWS CLI installed)
will look at the troubleshooting doc. Thank you for helping!
I’ve set aws:skipCredentialsValidation to true, and it’s still
error: Preview failed: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
If I also set
aws:skipMetadataApiCheck
to false, the same error happens
just installed the AWS CLI on the EC2 instance itself, and I can confirm that I can run
aws ec2 describe-instances --instance-ids <ID>
l

limited-rainbow-51650

08/03/2022, 1:54 PM
So, by now, how does your stack config file look? Could you paste it here?
r

rhythmic-branch-12845

08/03/2022, 1:54 PM
Copy code
config:
  aws:region: ap-southeast-1
  aws:skipCredentialsValidation: "true"
  #aws:skipMetadataApiCheck: "false"
l

limited-rainbow-51650

08/03/2022, 1:55 PM
And what is the outcome if you uncomment that last line?
r

rhythmic-branch-12845

08/03/2022, 2:00 PM
The same error :(
l

limited-rainbow-51650

08/03/2022, 2:04 PM
To summarize: • you have an EC2 instance with an instance profile attached • you are logged in on that EC2 instance and executing all of below commands on that machine • with the AWS CLI, you can successfully run an
aws describe-instances
command • You ran
pulumi config set aws:skipCredentialsValidation true
• You ran
pulumi config set aws:skipMetadataApiCheck false
pulumi preview
or
pulumi up
always results in
unable to validate AWS credentials
Is that correctly summarized?
r

rhythmic-branch-12845

08/03/2022, 2:16 PM
yes. Correction for the last point though: I’ve always been doing
pulumi up -r
. So: •
pulumi refresh
(or
pulumi up -r
) gives me
error: Preview failed: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
pulumi preview
and
pulumi up
(no refresh) gives me a new error:
Copy code
error: 1 error occurred:
        * error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.

    Please see <https://registry.terraform.io/providers/hashicorp/aws>
    for more information about providing credentials.

    Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or "AWS_EC2_METADATA_DISABLED" environment variable
ok what the heck. I just tried commenting out that last line in Pulumi.dev.yaml, and now
pulumi preview
and
pulumi up
work? any sort of “refresh” operation though,
pulumi refresh
, or
pulumi up -r
still does not work, with the same old error message
To summarize: • I have an EC2 instance with an instance profile attached • I am logged in on that EC2 instance and executing all of below commands on that machine • with the AWS CLI, I can successfully run an
aws describe-instances
command • I ran
pulumi config set aws:skipCredentialsValidation true
• I ran
pulumi config set aws:skipMetadataApiCheck false
pulumi preview
works •
pulumi up
works - and is even able to apply changes! • BUT
pulumi refresh
and
pulumi up -r
always result in
error: Preview failed: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
l

limited-rainbow-51650

08/03/2022, 2:37 PM
Next try:
pulumi config set aws:skipGetEc2Platforms false
r

rhythmic-branch-12845

08/03/2022, 2:38 PM
sorry before that I have an update. Was in the midst of typing it, but -> final update: I don’t know what’s going on, and it feels like AWS has been jerking me around, but… now everything works! I’m going to put it down to AWS being stupid or something; but thank you for staying with me on this. I learned a thing or two (or more) through this incident; and my confidence and knowledge of pulumi has increased (even if it took this ridiculous state of affairs)
💜 1
l

limited-rainbow-51650

08/03/2022, 2:47 PM
🎉 Well, it was bugging me as well. What I found out along the way, by reading a number of our own GH issues and pull requests is that all the different
skip…
options (see Installation & Configuration) had some of their default values changed to speed up the authentication process for the majority of the cases. The
aws:skipMetadataApiCheck
option needs to be set to
false
in CI cases where the underlying worker nodes are having an instance profile attached (your case).
r

rhythmic-branch-12845

08/03/2022, 2:50 PM
right! it will speed things up, but oh, it has been such a ride for those of us trying to use instance profiles! thank you so much, Ringo, for your help!!
one last note / test with
Pulumi.dev.yaml
just to test the config settings: it looks like I need both
pulumi config set aws:skipCredentialsValidation true
and
pulumi config set aws:skipMetadataApiCheck false
for things to work: with only
aws:skipMetadataApiCheck: "false"
I get
error: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
with only
aws:skipCredentialsValidation: "true"
I get
Copy code
error: 1 error occurred:
        * error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.

    Please see <https://registry.terraform.io/providers/hashicorp/aws>
    for more information about providing credentials.

    Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or "AWS_EC2_METADATA_DISABLED" environment variable
I have to say that I was not expecting to have to know stuff at this level and deal with these settings to be able to get things to work (once AWS got cooperative, that is) with an instance profile. But again, thank you.
l

limited-rainbow-51650

08/03/2022, 2:54 PM
Well, we clearly have some more work to do to make all of this easier. 😉
👍 1
🙏 1
Here is an issue you might want to subscribe to: https://github.com/pulumi/pulumi-aws/issues/1692
👌 1
60 Views