🐳 How do you usually build your Docker images ? 🐳 Do you use

pulumi-docker

or do you build your images externally ? We’ve been using

pulumi-docker

for about two years and there’s a ton of stuff that doesn’t work very well : log outputs, using buildx for multi-platform builds (AWS Graviton is on ARM arch), caching... Overall, it feels like we’re using

docker.Image

merely because we have to : otherwise we can’t get the build digest which uniquely identifies an image So I’m wondering if there are other ways : how can I build my image externally (e.g. using GitHub Action’s build and push), push it on ECR and yet let Pulumi know which exact version (tag, on ECR) I want to deploy in a given Lambda ?

Idea 1: You can use the Command provider to run Docker commands. If you want to run a command synchronously on every

pulumi up

, you can use

command.local.run()

, e.g to shell out to

make

. I'd probably start here since Make is already really good at keeping track of file dependencies. I believe you should be able to query for the digest using

docker image ls

once the image is built. https://www.pulumi.com/registry/packages/command/api-docs/local/run/

☝️ This is what we do. It's better to build your artifacts once and then deploy specific artifacts into different environments than to build them every time you deploy (possibly getting different artifacts, depending on how your images are set up), IMO.

It's better to build your artifacts once and then deploy specific artifacts into different environments than to build them every time you deploy

agree with this. but you can totally do this with Pulumi too

Thank you for your answers @stocky-restaurant-98004 @full-artist-27215 @steep-toddler-94095 I tend to agree with you, but I believe I’m stuck in a “egg or hen” kind of problem here, specific to AWS In order to deploy a lambda.Function resource with a container, you need to pass an

image_uri

pointing a valid image on ECR Building the image externally means using

aws update-function-configuration {function_name}

once the Docker image is built in order to notify Lambda that a new image is to be deployed... and that supposes that the Lambda function resource is deployed, which agains needs a valid ECR image to do so. How do you propose to solve this ? Do you use some kind of placeholder ECR image (and if so which) ? Or do you separate the stacks, which makes the overall process more complex ?

Wouldn't Josh's Idea #2 work? You can build the image in Github Actions outside of Pulumi in step #1, and then pass that image URI to step #2 which runs Pulumi up. If it's a new lambda Pulumi will create it with the provided image URI and if it's updating an existing lambda, it'll just update it with the specified image URI

I don't think it would Imagine I build my image, push it to ECR and tag it

v1

. I then pass that to my Pulumi config

tag:v1

and

pulumi up

. My Lambda gets initially deployed : all fine. Then I use GitHub Actions to build and push a

v2

on ECR, what next ? If I use the AWS CLI to update the Lambda's

image_uri

, the next

pulumi refresh

will say that the image is

v2

whereas the config still says

v1

so it'll get replaced in the next

pulumi up

Updating the config manually every time doesn't seem like a sustainable thing to do either : in practice, some of my dev will forget to do it and it'll create discrepancies. I try to minimise the amount of manual operations

Why do you need to update the image uri via the AWS CLI instead of in Pulumi? I'm not following that part.

Using Pulumi to build the docker image leads to very slow Pulumi up sometimes : a bad DX overall The logging is also quite bad : you don't really understand what's going on and why layers are rebuilding Also, if you need to use different architectures then Buildx simply isn't supported by Pulumi docker Overall, it feels like taking that complex process out of Pulumi (which is already complex) is a better practice in terms of simplicity and risk management But that like… my opinion 😎

Have you tried

command.local.Command

? I agree that

docker.Image

is pretty rough, especially in Python. We are aware and are looking at options for improving it.

Then I use GitHub Actions to build and push a

v2

on ECR, what next ?

you can pass that image URI into another github actions job/step that runs

pulumi up

against the lambda stack, no?

@bright-orange-69401 I'm on the same stack (AWS, ECR, Pulumi, GitHub Actions) and have struggled with building Docker images in Pulumi for a while too. If you are willing to sacrifice multi-arch builds, the rest works quite good with a bit of orchestration. Here's the relevant parts, if it helps. GHA workflow:

jobs:
  infra:
    name: Provision infrastructure
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: eu-central-1
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v2
        with:
          platforms: all
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          install: true
      - name: Cache Docker layers
        uses: actions/cache@v3
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-
      - name: Create or update stack resources
        uses: pulumi/actions@v3
        with:
          command: up
          refresh: true
          stack-name: *****
          work-dir: infra
          cloud-url: s3://*****
        env:
          PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
      - name: Move cache
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache

Pulumi program:

...
cache_from = "/tmp/.buildx-cache"
cache_to = (
    "/tmp/.buildx-cache-new" if os.getenv("GITHUB_ACTIONS") else cache_from
)

image = docker.Image(
    "*****",
    build=docker.DockerBuild(
        context="..",
        extra_options=[
            "--output=type=docker",
            f"--cache-from=type=local,src={cache_from}",
            f"--cache-to=type=local,mode=max,dest={cache_to}",
        ],
    ),
    image_name=repository.repository_url,
    registry=registry,
    skip_push=False,
)
...

Also, when it comes to the Docker build speed, I wouldn't blame Pulumi. What can take a significant amount of time is the cross-platform build using Buildx (e.g. building for ARM64 on x86-64 GitHub Actions runner). I've experienced this often, mainly with Node.js apps though.

Thanks @proud-art-41399 , it does help 😉 Have you tried using the GitHub Cache API in GitHub Actions ? https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md#github-cache

Not yet, but I plan to test it out.

@proud-art-41399, thank you for this! I was able to get an efficient setup all thanks to this example!

Glad it helped 🙂 Btw, we got to testing the GitHub Actions cache for Docker and it works pretty well – it seems to be a bit faster and simplifies the setup.

Yup, we’re moving on to that setup as well Still considering Bazel though… because we end up having quite a lot of dependencies

@proud-art-41399 @bright-orange-69401 First of all thanks for sharing these tips for speeding up the docker build. It's already very helpful. Can you please to provide some more details on how you used the

GitHub Actions cache

? Did you end up building the container separately instead of doing inside the Pulumi code? Or you are currently using this setup that you mentioned above?

Hi @agreeable-window-77899, I ended up using the Docker GHA cache which simplified the whole thing. I could remove these parts from the GitHub Actions workflow:

...
      - name: Cache Docker layers
        uses: actions/cache@v3
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-
      ...
      - name: Move cache
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache

as they are now not needed. And the relevant part of the Pulumi program now looks like this:

...
extra_options = ["--output=type=docker"]
if os.getenv("GITHUB_ACTIONS"):
    extra_options.extend(
        ["--cache-to=type=gha,mode=max", "--cache-from=type=gha"]
    )

image = docker.Image(
    "xxx",
    build=docker.DockerBuild(context="..", extra_options=extra_options),
    image_name=repository.repository_url,
    registry=registry,
    skip_push=False,
)
...

hi @proud-art-41399 Thanks for getting back. So did you add the

docker/build-push-action@v3

action instead of the

actions/cache@v3

then?

Nope, there's no need to, the push happens during the run of Pulumi program.

Thanks @proud-art-41399 That's very helpful 💐

Hey @proud-art-41399, I've switched to using

type=gha

and I can see the caches get created automatically! However, in my case, I am building two Docker Images (via

awsx.ecr.buildAndPushImage

), and I don't think the caching is working. From the detailed diagnostics on Pulumi Cloud, both images seem to be built from scratch, without looking at the cached layers. After the build, the new image layers are cached. Do you think there's some additional configuration required for multiple images?

I think there's some confusion here : there's no

awsx.ecr.buildAndPushImage

AFAIK There's just [awsx.ecr.Image](https://www.pulumi.com/registry/packages/awsx/api-docs/ecr/image/) If you use the

extra_options

like Thomas showed in the code snippet above then you should be fine... ?

Hey @bright-orange-69401, ah I actually haven't updated

pulumi

and related packages since 3 months ago. Still, the IaC script works flawlessly. My issue seems to be with the GitHub actions cache. I realized I'm not setting a scope for each image, which is what's probably causing the issue.