Hi folks. My lead eng is on vacation so I'm trying...
# aws
m
Hi folks. My lead eng is on vacation so I'm trying to figure this out and hopefully someone here can help. My
pulumi up
command is hanging and not responding to ctrl-C, ctrl-\, or any other commands I'd usually try to kill the process. I'm using a super basic stack configuration just to try to figure out what's going on:
Copy code
"""An AWS Python Pulumi program"""

import pulumi
from pulumi_aws import s3

# Create an AWS resource (S3 Bucket)
bucket = s3.BucketV2('my-bucket')

# Export the name of the bucket
pulumi.export('bucket_name', bucket.id)
And this is what I'm seeing on my screen after more than 1 hour of waiting. I'm assuming this is a configuration issue but shouldn't I get an error message or something? Why would it be locked up like this on such a basic implementation?
l
I don't know why it wouldn't respond to an interrupt or kill, unless maybe you're using a browser-based terminal that's not passing those signals on to the terminal? As to why it might be hanging like that: given what I can see on the terminal, my guess is that your stack uses short-lived sessions (e.g. via SSO) and the current session has ended. I don't know why the AWS SDK just hangs like that when dealing with expired short-lived sessions, but that's what it does for me. You may need to run
aws sso login
or some equivalent.
m
Hi @little-cartoon-10569 thanks for hopping in. I'm indeed using aws SSO, but I'm running
aws sso login
immediately before running
pulumi up
so I wouldn't expect that to be the issue. Also just running in iTerm on my macbook.
I'm running this command to try to see what it might be hanging on but nothing is obvious:
Copy code
TF_LOG=TRACE pulumi up --logtostderr --logflow -v=10 2> out.txt
Happy to share the output of that file if it'll help
l
Have you confirmed that your stack config is using the short-lived session? For example, if it is hard-coded to use profile X and you're using the default profile, then the short-lived session won't be picked up. You'd have to run
aws --profile X sso login
to get it to work in that case.
m
Right, I'm running
aws sso login --profile my-profile
. I just created this stack - where would I look to check if it's using a short-lived session?
Maybe I have to
export AWS_PROFILE=my-profile
before running
pulumi up
?
l
In your Pulumi.yaml, Pulumi.<stack>.yaml, or your AWS provider constructor in code. You're looking for the AWS provider config: is it specifying a profile? Or maybe the env vars?
m
Yeah I'm trying literally with the base aws-python template just to narrow it down. Here's my Pulumi.yaml:
Copy code
name: test3
description: A minimal AWS Python Pulumi program
runtime:
  name: python
  options:
    toolchain: pip
    virtualenv: venv
config:
  pulumi:tags:
    value:
      pulumi:template: aws-python
l
No, you don't need the env var, unless the provider is explicitly referring to it. If it's the default provider or falling back via the normal precendence, then simply specifying the prlfile should be good.
There's nothing in there about aws. You'd be looking for
Copy code
aws:
  profile: X
in your stack file, or
Copy code
config:
  aws:
    profile: X
in your project file.
m
Just this:
Copy code
config:
  aws:region: us-east-1
I think you're right that it's something to do with the session. Wondering if I should try to grab an access key and secret key and try with those instead
l
Right. Is the AWS provider constructed in your project code? If not, then you're using the default profile. In which case, try
aws sso login
again, without the
--profile
.
m
But I'm confused to why it would hang as opposed to saying it can't access s3 or whatever the first error would be
l
Where does it say that?
m
Aha, ok the provider is not explicit in my project code. That might be what I'm missing
l
It's ok to use the default provider. I don't, but it does work. Just don't specify a profile when you're logging in, and it should be good.
m
Well my default provider on my machine is using creds that no longer exist, that must be what's going on. Let me see about specifying the profile explicitly in my code
OK, so I actually don't have a default profile on this machine, they're all named.
let me try adding a default one and see if that works
l
Yea, you'll need something to map your sso_session to the default provider, else it won't be looking in the right place.
However, since there's no sso_session currently mapped, the Pulumi code isn't using short-lived sessions, and that invalidates my assumptions about the observed behaviour. Normal creds should be been found to be missing, and that works better with the AWS SDK. It gives real error messages.
Well, it used to, last time I used ACCESS_KEY and SECRET_KEY without a session ID. Which might have been 2022.
m
Yeah it's been a while for me too
No dice on the default profile either. No error logs that I can see. Just seems like it's hanging even before attempting to do anything with AWS.
And with CTRL-C not working it's like one of the threads is locked up and not listening to whatever channel is sending signals.
Gonna try one last hail mary with access keys
nope. 🤷
I don't understand how this could be happening with no errors and nothing in the debug logs despite high verbosity.
@little-cartoon-10569 any recommendations on other places I could go to get help? like pulumi consultants that have an hourly rate?
l
No sorry, I've never looked into that.
Can you use the AWS CLI to access anything?
aws s3 ls
or similar?
m
yep, that’s all working fine
l
I suppose
pulumi login
might be worth a try too?
m
I looked into it some more online - I have an M1 mac, but the pulumi binary and aws provider binary is x86_64, which I’m guessing is because I have rosetta 2 enabled. I did all this a few years ago back when not everything supported the M1 architecture yet. Maybe this has something to do with it?
l
You're outside my area of expertise now. Worth a shot, re-install Pulumi for Apple hardware
m
That was it
@little-cartoon-10569 thanks for hanging out
q
I'm curious to learn how you discovered that you're running x86_64 versions of
pulumi
and the AWS provider. We've been thinking about adding detection mechanisms for this right into pulumi: https://github.com/pulumi/pulumi/issues/16598
m
@quick-house-41860 so here’s what I did: • I had an older version of Pulumi installed from a few years ago. When I went to run it, it prompted me to upgrade (using homebrew) • Went through that process above, then decided to remove Pulumi via homebrew and install it via the script at get.pulumi.com. Ran into the same issue w/ hanging after it installed the aws provider. • Seems like that also installed it (both the pulumi binary and aws provider) as x86. The way I found out was I just ran
which pulumi
, navigated to that folder, and then ran
file pulumi
and it told me it was x86. I then went into the plugins folder and did the same thing and saw that it was also x86. • From there I just removed the entire
~/.pulumi
folder, then I downloaded that script from get.pulumi.com and added some print lines to where it figured out the arch. • Unclear what I did to get it to work, but when I ran it locally using
zsh
instead of what’s in your documentation (
sh
), it installed correctly as arm64 and everything started working.
If I were you guys I’d be mostly worried about why the installation decided to use x86 vs arm depending on my shell. This is an older laptop (from 2021 I think) and I did have Rosetta running at one point for backward compatibility, so it might be some remnant of that.
q
Thanks! I wonder if you've changed the arch of
sh
to x86 🤔 I remember that this was the workaround for some tools when the apple silicon macbooks first came out.
m
That sounds like the likely culprit. I did just check what architecture my
sh
command thinks it’s running in and it’s giving me arm64, but I’m not entirely sure how all of that works with Rosetta.
Copy code
jason@Jasons-MacBook-Pro-2 datahub % /bin/sh
sh-3.2$ uname -a
Darwin Jasons-MacBook-Pro-2.local 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:03:15 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T6000 arm64