Hello all Thank you for you time reading this! Sorry if the question is a bit vague, I hope it's sufficiently clear... I've encountered a problem with a pulumi stack using

pulumi-docker

to deploy images on an ECR repository; the

pulumi up

command works on my laptop but not on my colleague one; we tried to build the image locally and also pushing it to the ECR repo using the commands suggested in the AWS documentation, but the build/push fails when run through pulumi. this is the error message:

(...)
error: error reading build output: failed to read downloaded context: failed to load cache key: invalid response status 403

this is the code in the pulumi main script:

ref_admin = pulumi.StackReference(f"{org}/admin/{stack}")
service_assets = ref_admin.get_output("ecs")

image_repo = service_assets['ecrs']['myservice']['repo']
image_name = image_repo.apply(lambda val: f"{val}:{get_docker_tag()}")
image = docker.Image(
    "myservice-image",
    build=docker.DockerBuildArgs(
        context="..",  # Path to the build context
        platform="linux/amd64",
    ),
    image_name=image_name,
    registry=docker.RegistryArgs(
        server=image_repo,
        username=aws.ecr.get_authorization_token().user_name,
        password=aws.ecr.get_authorization_token().password,
    ),
)

I'm not quite sure what the problem is... I tried to add extra logging with

--logtostderr --logflow -v=10 2> out.txt

, and I can see the token being passed:

(...)
17:43:57.730951   93774 eventsink.go:59] I0203 17:43:57.730430   93845 rpc.go:292] Unmarshaling property for RPC[]: registry={map[password:{====REDACTED-TOKEN====} server:{123REDACTED456.dkr.ecr.us-east-1.amazonaws.com/myservice} username:{AWS}]}
(...)

I can also see these lines that may suggest some other error:

I0203 17:43:58.687476   93774 eventsink.go:70] eventSink::Info(<{%reset%}>digest: sha256:63fd8b53c2e09b89dc1972305eaf0139e403ef05cb750de076299726b4c1881a
[internal] load remote build context
error: invalid response status 403<{%reset%}>)
I0203 17:43:58.687678   93774 eventsink.go:59] time="2025-02-03T17:43:58Z" level=error msg="Can't add file /.../.venv/bin/ruff to tar: io: read/write on closed pipe"

I'm not sure if this is related at all as

ruff

is not used by the

Dockerfile

and running

docker build ..

works as expected... and the error mention a

403

error which looks like an HTTP forbidden error code... any help would be much appreciated...

Are you both logged into ECR with the same credentials? It looks like you have access to the repository and the other person doesn't.

we tried the following on his MAC terminal and it all worked:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 123REDACTED.dkr.ecr.us-east-1.amazonaws.com

docker build -t myservice ..

docker tag myservice:latest 123REDACTED.dkr.ecr.us-east-1.amazonaws.com/myservice:latest

docker push 123REDACTED.dkr.ecr.us-east-1.amazonaws.com/myservice:latest

those are the commands that AWS suggests when one navigate to the ECR repo and click on "Show push commands" button

And how are your AWS providers configured for your Pulumi code? Are they logging in as the IAM users / roles you're testing with?

hi tenwit the aws creds are passed as access/secret key pairs (they're set in the

/aws/credentials

file) and the

aws.ecr.get_authorization_token()

seem to work correctly...

As far as I understand, the Terraform Docker provider that's wrapped by the Pulumi provider is a client for the Docker API (similar to the

docker

CLI command). Are you running the same Docker version on your machines? A cursory Google search suggests that the errors you see are all somehow related to BuildKit, e.g., https://github.com/moby/buildkit/issues/5623

yes we did check that... my version is very old while his is a lot more recent... we also checked the docker desktop is running on his laptop... really weird...

Then I think this is your smoking gun.

??

my version is very old while his is a lot more recent...

Since Docker 23.0, BuildKit is the default builder for Docker Desktop and Docker Engine

yeah, on my laptop it works... it his, using the much more recent version, that fails to build...

Based on that, I think that your problem is not AWS but the communication between the Docker provider and your local Docker API

so you suggests he downgraders to the same version as mine? fyi:

$ docker --version
Docker version 24.0.7, build afdd53b

What versions are you running exactly?

☝️ 😄

And you?

errr. sorry, that's mine... let see if i took a note... just a sec

so you suggests he downgraders to the same version as mine?

You could try that but I don't think that's a good solution. I'd look into https://github.com/moby/buildkit/issues/5623 and see if your colleague's setup matches the configuration described in the issue

ok, that'll have to wait, 11pm here! thanks for the help, hopefully that's the case 🤞 👍

This is also related to the communication with the local Docker API

ok thanks a lot!

I think if you run

find .

in the directory you're building from, it's the first file that's not ignored by Docker that shows up. (You should probably add your virtualenv to

.dockerignore

but then you'll just get the next-best file in the error message.)

yep, good point!

Good luck debugging this tomorrow. Once you figure it out, I'd be curious to learn what the root cause was 🙂

yeah... sure!

Yes, but it's the Docker CLI in a specific terminal environment that's communicating with the API, not the Pulumi/Terraform provider

a-ha!!! https://github.com/pulumi/pulumi-docker/issues/967

My issue was different but I wanted to say thanks because you also helped me fix an issue with building on an ARM mac to push to ECR