https://pulumi.com logo
Title
b

bright-orange-69401

09/15/2022, 6:42 AM
🐳 How do you usually build your Docker images ? 🐳 Do you use
pulumi-docker
or do you build your images externally ? We’ve been using
pulumi-docker
for about two years and there’s a ton of stuff that doesn’t work very well : log outputs, using buildx for multi-platform builds (AWS Graviton is on ARM arch), caching... Overall, it feels like we’re using
docker.Image
merely because we have to : otherwise we can’t get the build digest which uniquely identifies an image So I’m wondering if there are other ways : how can I build my image externally (e.g. using GitHub Action’s build and push), push it on ECR and yet let Pulumi know which exact version (tag, on ECR) I want to deploy in a given Lambda ?
s

stocky-restaurant-98004

09/15/2022, 12:59 PM
Idea 1: You can use the Command provider to run Docker commands. If you want to run a command synchronously on every
pulumi up
, you can use
command.local.run()
, e.g to shell out to
make
. I'd probably start here since Make is already really good at keeping track of file dependencies. I believe you should be able to query for the digest using
docker image ls
once the image is built. https://www.pulumi.com/registry/packages/command/api-docs/local/run/
Idea 2: You can build the image in some other pipeline step, and when that's done, pass the tag in as a required config value to your Pulumi program: https://www.pulumi.com/docs/intro/concepts/config/#code
f

full-artist-27215

09/15/2022, 2:34 PM
☝️ This is what we do. It's better to build your artifacts once and then deploy specific artifacts into different environments than to build them every time you deploy (possibly getting different artifacts, depending on how your images are set up), IMO.
s

steep-toddler-94095

09/16/2022, 2:20 AM
It's better to build your artifacts once and then deploy specific artifacts into different environments than to build them every time you deploy
agree with this. but you can totally do this with Pulumi too
b

bright-orange-69401

09/16/2022, 7:07 AM
Thank you for your answers @stocky-restaurant-98004 @full-artist-27215 @steep-toddler-94095 I tend to agree with you, but I believe I’m stuck in a “egg or hen” kind of problem here, specific to AWS In order to deploy a lambda.Function resource with a container, you need to pass an
image_uri
pointing a valid image on ECR Building the image externally means using
aws update-function-configuration {function_name}
once the Docker image is built in order to notify Lambda that a new image is to be deployed... and that supposes that the Lambda function resource is deployed, which agains needs a valid ECR image to do so. How do you propose to solve this ? Do you use some kind of placeholder ECR image (and if so which) ? Or do you separate the stacks, which makes the overall process more complex ?
s

steep-toddler-94095

09/16/2022, 6:25 PM
Wouldn't Josh's Idea #2 work? You can build the image in Github Actions outside of Pulumi in step #1, and then pass that image URI to step #2 which runs Pulumi up. If it's a new lambda Pulumi will create it with the provided image URI and if it's updating an existing lambda, it'll just update it with the specified image URI
b

bright-orange-69401

09/17/2022, 5:39 AM
I don't think it would Imagine I build my image, push it to ECR and tag it
v1
. I then pass that to my Pulumi config
tag:v1
and
pulumi up
. My Lambda gets initially deployed : all fine. Then I use GitHub Actions to build and push a
v2
on ECR, what next ? If I use the AWS CLI to update the Lambda's
image_uri
, the next
pulumi refresh
will say that the image is
v2
whereas the config still says
v1
so it'll get replaced in the next
pulumi up
Updating the config manually every time doesn't seem like a sustainable thing to do either : in practice, some of my dev will forget to do it and it'll create discrepancies. I try to minimise the amount of manual operations
Maybe the best way would be to use a placeholder
image_uri
such as
public.ecr.aws/lambda/python:3.9
then use
ignore_changes=["image_uri"]
so that Pulumi doesn't replace if the value gets changed externally
s

stocky-restaurant-98004

09/17/2022, 2:11 PM
Why do you need to update the image uri via the AWS CLI instead of in Pulumi? I'm not following that part.
b

bright-orange-69401

09/17/2022, 5:58 PM
Using Pulumi to build the docker image leads to very slow Pulumi up sometimes : a bad DX overall The logging is also quite bad : you don't really understand what's going on and why layers are rebuilding Also, if you need to use different architectures then Buildx simply isn't supported by Pulumi docker Overall, it feels like taking that complex process out of Pulumi (which is already complex) is a better practice in terms of simplicity and risk management But that like… my opinion 😎
s

stocky-restaurant-98004

09/17/2022, 9:36 PM
Have you tried
command.local.Command
? I agree that
docker.Image
is pretty rough, especially in Python. We are aware and are looking at options for improving it.
s

steep-toddler-94095

09/18/2022, 1:09 AM
Then I use GitHub Actions to build and push a
v2
on ECR, what next ?
you can pass that image URI into another github actions job/step that runs
pulumi up
against the lambda stack, no?
p

proud-art-41399

09/29/2022, 6:13 AM
@bright-orange-69401 I'm on the same stack (AWS, ECR, Pulumi, GitHub Actions) and have struggled with building Docker images in Pulumi for a while too. If you are willing to sacrifice multi-arch builds, the rest works quite good with a bit of orchestration. Here's the relevant parts, if it helps. GHA workflow:
jobs:
  infra:
    name: Provision infrastructure
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: eu-central-1
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v2
        with:
          platforms: all
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          install: true
      - name: Cache Docker layers
        uses: actions/cache@v3
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-
      - name: Create or update stack resources
        uses: pulumi/actions@v3
        with:
          command: up
          refresh: true
          stack-name: *****
          work-dir: infra
          cloud-url: s3://*****
        env:
          PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
      - name: Move cache
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache
Pulumi program:
...
cache_from = "/tmp/.buildx-cache"
cache_to = (
    "/tmp/.buildx-cache-new" if os.getenv("GITHUB_ACTIONS") else cache_from
)

image = docker.Image(
    "*****",
    build=docker.DockerBuild(
        context="..",
        extra_options=[
            "--output=type=docker",
            f"--cache-from=type=local,src={cache_from}",
            f"--cache-to=type=local,mode=max,dest={cache_to}",
        ],
    ),
    image_name=repository.repository_url,
    registry=registry,
    skip_push=False,
)
...
Also, when it comes to the Docker build speed, I wouldn't blame Pulumi. What can take a significant amount of time is the cross-platform build using Buildx (e.g. building for ARM64 on x86-64 GitHub Actions runner). I've experienced this often, mainly with Node.js apps though.
b

bright-orange-69401

09/29/2022, 1:05 PM
Thanks @proud-art-41399 , it does help 😉 Have you tried using the GitHub Cache API in GitHub Actions ? https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md#github-cache
p

proud-art-41399

09/29/2022, 1:36 PM
Not yet, but I plan to test it out.
m

mysterious-apartment-62241

10/20/2022, 5:55 PM
@proud-art-41399, thank you for this! I was able to get an efficient setup all thanks to this example!
p

proud-art-41399

10/20/2022, 6:02 PM
Glad it helped 🙂 Btw, we got to testing the GitHub Actions cache for Docker and it works pretty well – it seems to be a bit faster and simplifies the setup.
b

bright-orange-69401

10/20/2022, 6:03 PM
Yup, we’re moving on to that setup as well Still considering Bazel though… because we end up having quite a lot of dependencies
a

agreeable-window-77899

11/17/2022, 10:57 AM
@proud-art-41399 @bright-orange-69401 First of all thanks for sharing these tips for speeding up the docker build. It's already very helpful. Can you please to provide some more details on how you used the
GitHub Actions cache
? Did you end up building the container separately instead of doing inside the Pulumi code? Or you are currently using this setup that you mentioned above?
p

proud-art-41399

11/17/2022, 11:36 AM
Hi @agreeable-window-77899, I ended up using the Docker GHA cache which simplified the whole thing. I could remove these parts from the GitHub Actions workflow:
...
      - name: Cache Docker layers
        uses: actions/cache@v3
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-
      ...
      - name: Move cache
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache
as they are now not needed. And the relevant part of the Pulumi program now looks like this:
...
extra_options = ["--output=type=docker"]
if os.getenv("GITHUB_ACTIONS"):
    extra_options.extend(
        ["--cache-to=type=gha,mode=max", "--cache-from=type=gha"]
    )

image = docker.Image(
    "xxx",
    build=docker.DockerBuild(context="..", extra_options=extra_options),
    image_name=repository.repository_url,
    registry=registry,
    skip_push=False,
)
...
a

agreeable-window-77899

11/17/2022, 12:02 PM
hi @proud-art-41399 Thanks for getting back. So did you add the
docker/build-push-action@v3
action instead of the
actions/cache@v3
then?
p

proud-art-41399

11/17/2022, 12:03 PM
Nope, there's no need to, the push happens during the run of Pulumi program.
a

agreeable-window-77899

11/17/2022, 4:16 PM
Thanks @proud-art-41399 That's very helpful 💐
m

mysterious-apartment-62241

12/23/2022, 8:07 AM
Hey @proud-art-41399, I've switched to using
type=gha
and I can see the caches get created automatically! However, in my case, I am building two Docker Images (via
awsx.ecr.buildAndPushImage
), and I don't think the caching is working. From the detailed diagnostics on Pulumi Cloud, both images seem to be built from scratch, without looking at the cached layers. After the build, the new image layers are cached. Do you think there's some additional configuration required for multiple images?
For more context: • not using the
qemu
action • the issue seems to occur with / without
refresh: true
for the
pulumi/push
action
b

bright-orange-69401

12/23/2022, 9:12 AM
I think there's some confusion here : there's no
awsx.ecr.buildAndPushImage
AFAIK There's just [awsx.ecr.Image](https://www.pulumi.com/registry/packages/awsx/api-docs/ecr/image/) If you use the
extra_options
like Thomas showed in the code snippet above then you should be fine... ?
m

mysterious-apartment-62241

12/23/2022, 9:30 AM
Hey @bright-orange-69401, ah I actually haven't updated
pulumi
and related packages since 3 months ago. Still, the IaC script works flawlessly. My issue seems to be with the GitHub actions cache. I realized I'm not setting a scope for each image, which is what's probably causing the issue.
Thank you