https://pulumi.com logo
Join Slack
Powered by
can Pulumi be run from an EC2 instance that has an...
# aws
r
can Pulumi be run from an EC2 instance that has an instance profile assigned to it? or must we set up AWS credentials?
s
you should be able to run it from any system which is api authenticated.
l
You should consider using a role that you assume, but you don't need extra credentials. The instance profile is exposed within the instance via the access and secret key env vars, and Pulumi will use those if no other creds (e.g. default profile) are set up.
r
I am expecting to use an instance profile, which basically assigns a role to the EC2 instance itself. This is not exposed as your standard env vars, though (that is the point, to prevent having to deal with env vars). My understanding is that anything that uses the AWS SDK will be able to work with this setup, but it looks like Pulumi does not use the SDK? This what I get in the EC2 instance, without any “AWS_*” env vars:
Copy code
Diagnostics:
  aws:ec2:Instance (api):
    error: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
l
As I understand it, the instance profile is available through the EC2 metadata, and it's the lowest priority source of creds. I'm not certain that Pulumi's AWS provider uses it. Therefore, when I run from EC2, I've always pulled the instance profile values out to ACCESS and SECRET env vars, and it's worked without a hitch.
You can grab the values by wgetting / curling this URL, and piping through jq or similar: http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance
r
thank you. That’s helpful to know! I’m expecting Pulumi to do better, but I guess until it does, this is how I will have to get this working. Thank you!
this was working for a while.. but I have 2 issues right now: 1. this first one is the “obvious” one: it looks like to survive with using the instance profile BUT without having the underlying logic of the SDK to help me out, I would basically have to duplicate this aws credential pulling method myself every single time I call 
pulumi up
. That’s because these credentials aren’t permanent, and have a timeout 2. leaving number 1 aside, it was working for a while for me… but after changing the permissions of the IAM role (it currently allows for 
"*", "*"
), I am getting 
AuthFailure: AWS was not able to validate the provided access credentials
and Pulumi no longer works?
it appears that there might be some bug with using temporary creds or something. No matter what I did, i was not able to get this resolved. I had to switch to using non-temporary creds, and then everything was ok 😠
l
Did you set the session token too? You need 3 env vars for temp creds.
You don't need to duplicate the cred-pulling method if you use an assumed role for Pulumi, which is what I recommend. You can create a profile that assumes role X and uses the default profile as its source (configure this in ~/.aws/config). Since the default profile uses the instance profile, you don't need to do anything else: so long as role X can be assumed by the instance profile role, it works every time.
It's also a lot safer: if an attacker gains access to your EC2 instance and the instance profile gives them Pulumi-like powers, they can destroy your infrastructure. If you use the role-assumption requirement, they don't have those powers. They need to know that they have to assume the extra role before they get those powers.
r
I did set the session token, yes. It was working all along until somehow it didn’t anymore. Sorry, but I’m somewhat confused. When I create my instance profile, I already tie it to the role that I want, and this role has the required permissions. Could you give an example for ~/.aws/config? is there anything I need to do with or within Pulumi to assume this role?
I’m trying the following in ~/.aws/config (and nothing in credentials):
Copy code
[default]
role_arn = arn:aws:iam::<ACCOUNT_ID>:role/pulumi
And then 
pulumi config set --path aws:assumeRole.roleArn arn:aws:iam::<ACCOUNT_ID>:role/pulumi
. No luck: 
error: Preview failed: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
l
While you can do all this in Pulumi, I suggest doing the role assumption in the AWS config file. Role assumption in config files is introduced here: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html. Look for uses of source_profile. In your config file, you can set up a profile that uses the default profile (= instance profile) as its source_profile. Your Pulumi provider can refer to the name of that profile. There isn't any particular advantage to assuming the role in the config file other than convenience: when I run my code locally, I have a simple profile with admin privs for my dev AWS account, and I don't need to do anything special in my Pulumi code to detect whether it needs to assume a role (in prod) or not (in dev). So long as my simple profile has the same name as the nice assumed-role profile, Pulumi can't tell the difference and it just works.
r
it sounds like what you’re saying is to do this? ~/.aws/config:
Copy code
[pulumi]
role_arn = arn:aws:iam::<ACCOUNT_ID>:role/pulumi
source_profile = default
(no need for ~/.aws/credentials) Pulumi.dev.yaml:
Copy code
config:
  aws:region: ap-southeast-1
  aws:profile: pulumi
With this setup, I still get the dreaded 
error: Preview failed: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
Is my setup correct?
I’ve also tried this in ~/.aws/config, with the same error:
Copy code
[pulumi]
role_arn = arn:aws:iam::<ACCOUNT_ID>:role/pulumi
credential_source = Ec2InstanceMetadata
l
@rhythmic-branch-12845 if you have an EC2 instance with an instance profile attached, you only have to specify the AWS region via an env variable or stack config. The AWS SDK for Go is integrated in both the AWS Classic (bridged from the TF AWS provider) and our new AWS Native provider. This SDK, like any other AWS SDK, implements the standard credential lookup like this: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials With an instance profile attached, the credentials are not exposed as env variables, but these are retrieved by the SDK on the EC2 metadata service, which is the URL posted by @little-cartoon-10569 as a previous answer in this thread. You don’t have to do this manually.
r
right. This is what I’m expecting: “config-free setup”. But it doesn’t seem to work as I expect. And given the error messages and the pointing to https://pulumi.io/install/aws.html where it’s all about setting up credentials, it’s made me wonder. NOTE: I am not using Go, but Python
l
This info is more aimed towards your local workstation. I’ll make sure that the EC2 instance profile variant get’s added to that. Now, unrelated to the programming language you choose to implement your infrastructure code, our providers are implemented in Go, so that is why I referred to the AWS SDK for Go. It is the AWS provider binary which executes the AWS authentication.
r
ok gotcha. Thank you for the clarification. If this is the case then I am truly stuck. Because I already have the instance profile attached to my EC2 instance. And until I use credentials, I am not able to get Pulumi working
l
@rhythmic-branch-12845 ok, let me dig further on this for you.
🙏 1
👍 1
@rhythmic-branch-12845 via the Github issues of our aws provider, I bumped into this link: https://github.com/pulumi/pulumi-aws#authenticating-pulumi-aws-via-ec2-instance-metadata So I think you will have to re-enable the use of EC2 metadata by setting this in your stack configuration:
Copy code
pulumi config set aws:skipMetadataApiCheck false
r
this looks promising, and maybe there is something to it. I’m not there yet. So this is my Pulumi.dev.yaml:
Copy code
config:
  aws:region: ap-southeast-1
  aws:skipMetadataApiCheck: "false"
And I’ve got absolutely nothing in ~/.aws now. And I still get 
error: Preview failed: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
l
I assume you are logged in on that EC2 instance at the moment? If so, can you try this: 
curl <http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance>
Do you get valid credentials back in the returned JSON document?
r
yes, I am logged in. The curl works and returns credentials.
l
I don’t know what permissions the EC2 instance have via the instance profile and attached role(s). Can you run a simple AWS cli command to test? 
aws ec2 describe-instances --instance-ids <your-instance-id>
r
not sure if this’ll give you what you’re looking for, but I’ll DM you
l
@rhythmic-branch-12845 I wasn’t really interested in the output, but the AWS CLI uses the same credential lookup strategy as the provider, given it is based on the same AWS SDK. If running that command with the AWS CLI works, it should work for Pulumi as well.
Can you perform these troubleshooting actions on your EC2 instance? https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_iam-ec2.html
@rhythmic-branch-12845 also, can you try to set this on your stack config: 
pulumi config set aws:skipCredentialsValidation true
r
re https://pulumi-community.slack.com/archives/CRH5ENVDX/p1659529800534399?thread_ts=1659084774.882929&amp;cid=CRH5ENVDX: sorry, I thought you wanted the output! this wasn’t run on the instance itself (no AWS CLI installed)
will look at the troubleshooting doc. Thank you for helping!
I’ve set aws:skipCredentialsValidation to true, and it’s still 
error: Preview failed: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
If I also set 
aws:skipMetadataApiCheck
to false, the same error happens
just installed the AWS CLI on the EC2 instance itself, and I can confirm that I can run 
aws ec2 describe-instances --instance-ids <ID>
l
So, by now, how does your stack config file look? Could you paste it here?
r
Copy code
config:
  aws:region: ap-southeast-1
  aws:skipCredentialsValidation: "true"
  #aws:skipMetadataApiCheck: "false"
l
And what is the outcome if you uncomment that last line?
r
The same error :(
l
To summarize: • you have an EC2 instance with an instance profile attached • you are logged in on that EC2 instance and executing all of below commands on that machine • with the AWS CLI, you can successfully run an 
aws describe-instances
command • You ran 
pulumi config set aws:skipCredentialsValidation true
• You ran 
pulumi config set aws:skipMetadataApiCheck false
• 
pulumi preview
or 
pulumi up
always results in 
unable to validate AWS credentials
Is that correctly summarized?
r
yes. Correction for the last point though: I’ve always been doing 
pulumi up -r
. So: • 
pulumi refresh
(or 
pulumi up -r
) gives me 
error: Preview failed: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
• 
pulumi preview
and 
pulumi up
(no refresh) gives me a new error:
Copy code
error: 1 error occurred:
        * error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.

    Please see <https://registry.terraform.io/providers/hashicorp/aws>
    for more information about providing credentials.

    Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or "AWS_EC2_METADATA_DISABLED" environment variable
ok what the heck. I just tried commenting out that last line in Pulumi.dev.yaml, and now 
pulumi preview
and 
pulumi up
work? any sort of “refresh” operation though, 
pulumi refresh
, or 
pulumi up -r
still does not work, with the same old error message
To summarize: • I have an EC2 instance with an instance profile attached • I am logged in on that EC2 instance and executing all of below commands on that machine • with the AWS CLI, I can successfully run an 
aws describe-instances
command • I ran 
pulumi config set aws:skipCredentialsValidation true
• I ran 
pulumi config set aws:skipMetadataApiCheck false
• 
pulumi preview
works • 
pulumi up
works - and is even able to apply changes! • BUT 
pulumi refresh
and 
pulumi up -r
always result in 
error: Preview failed: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
l
Next try: 
pulumi config set aws:skipGetEc2Platforms false
r
sorry before that I have an update. Was in the midst of typing it, but -> final update: I don’t know what’s going on, and it feels like AWS has been jerking me around, but… now everything works! I’m going to put it down to AWS being stupid or something; but thank you for staying with me on this. I learned a thing or two (or more) through this incident; and my confidence and knowledge of pulumi has increased (even if it took this ridiculous state of affairs)
💜 1
l
🎉 Well, it was bugging me as well. What I found out along the way, by reading a number of our own GH issues and pull requests is that all the different 
skip…
options (see Installation & Configuration) had some of their default values changed to speed up the authentication process for the majority of the cases. The 
aws:skipMetadataApiCheck
option needs to be set to 
false
in CI cases where the underlying worker nodes are having an instance profile attached (your case).
r
right! it will speed things up, but oh, it has been such a ride for those of us trying to use instance profiles! thank you so much, Ringo, for your help!!
one last note / test with 
Pulumi.dev.yaml
just to test the config settings: it looks like I need both 
pulumi config set aws:skipCredentialsValidation true
and 
pulumi config set aws:skipMetadataApiCheck false
for things to work: with only 
aws:skipMetadataApiCheck: "false"
I get 
error: unable to validate AWS credentials - see <https://pulumi.io/install/aws.html> for details on configuration
with only 
aws:skipCredentialsValidation: "true"
I get
Copy code
error: 1 error occurred:
        * error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.

    Please see <https://registry.terraform.io/providers/hashicorp/aws>
    for more information about providing credentials.

    Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or "AWS_EC2_METADATA_DISABLED" environment variable
I have to say that I was not expecting to have to know stuff at this level and deal with these settings to be able to get things to work (once AWS got cooperative, that is) with an instance profile. But again, thank you.
l
Well, we clearly have some more work to do to make all of this easier. 😉
👍 1
🙏 1
Here is an issue you might want to subscribe to: https://github.com/pulumi/pulumi-aws/issues/1692
👌 1
279 Views
Open in Slack
PreviousNext
Powered by