bright-orange-69401
09/15/2022, 6:42 AMpulumi-docker
or do you build your images externally ?
Weāve been using pulumi-docker
for about two years and thereās a ton of stuff that doesnāt work very well : log outputs, using buildx for multi-platform builds (AWS Graviton is on ARM arch), caching...
Overall, it feels like weāre using docker.Image
merely because we have to : otherwise we canāt get the build digest which uniquely identifies an image
So Iām wondering if there are other ways : how can I build my image externally (e.g. using GitHub Actionās build and push), push it on ECR and yet let Pulumi know which exact version (tag, on ECR) I want to deploy in a given Lambda ?stocky-restaurant-98004
09/15/2022, 12:59 PMpulumi up
, you can use command.local.run()
, e.g to shell out to make
. I'd probably start here since Make is already really good at keeping track of file dependencies.
I believe you should be able to query for the digest using docker image ls
once the image is built.
https://www.pulumi.com/registry/packages/command/api-docs/local/run/full-artist-27215
09/15/2022, 2:34 PMsteep-toddler-94095
09/16/2022, 2:20 AMIt's better to build your artifacts once and then deploy specific artifacts into different environments than to build them every time you deployagree with this. but you can totally do this with Pulumi too
bright-orange-69401
09/16/2022, 7:07 AMimage_uri
pointing a valid image on ECR
Building the image externally means using aws update-function-configuration {function_name}
once the Docker image is built in order to notify Lambda that a new image is to be deployed... and that supposes that the Lambda function resource is deployed, which agains needs a valid ECR image to do so.
How do you propose to solve this ? Do you use some kind of placeholder ECR image (and if so which) ? Or do you separate the stacks, which makes the overall process more complex ?steep-toddler-94095
09/16/2022, 6:25 PMbright-orange-69401
09/17/2022, 5:39 AMv1
. I then pass that to my Pulumi config tag:v1
and pulumi up
. My Lambda gets initially deployed : all fine.
Then I use GitHub Actions to build and push a v2
on ECR, what next ?
If I use the AWS CLI to update the Lambda's image_uri
, the next pulumi refresh
will say that the image is v2
whereas the config still says v1
so it'll get replaced in the next pulumi up
Updating the config manually every time doesn't seem like a sustainable thing to do either : in practice, some of my dev will forget to do it and it'll create discrepancies. I try to minimise the amount of manual operationsimage_uri
such as public.ecr.aws/lambda/python:3.9
then use ignore_changes=["image_uri"]
so that Pulumi doesn't replace if the value gets changed externallystocky-restaurant-98004
09/17/2022, 2:11 PMbright-orange-69401
09/17/2022, 5:58 PMstocky-restaurant-98004
09/17/2022, 9:36 PMcommand.local.Command
? I agree that docker.Image
is pretty rough, especially in Python. We are aware and are looking at options for improving it.steep-toddler-94095
09/18/2022, 1:09 AMThen I use GitHub Actions to build and push ayou can pass that image URI into another github actions job/step that runson ECR, what next ?v2
pulumi up
against the lambda stack, no?proud-art-41399
09/29/2022, 6:13 AMjobs:
infra:
name: Provision infrastructure
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
aws-region: eu-central-1
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
with:
platforms: all
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
with:
install: true
- name: Cache Docker layers
uses: actions/cache@v3
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-
- name: Create or update stack resources
uses: pulumi/actions@v3
with:
command: up
refresh: true
stack-name: *****
work-dir: infra
cloud-url: s3://*****
env:
PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
- name: Move cache
run: |
rm -rf /tmp/.buildx-cache
mv /tmp/.buildx-cache-new /tmp/.buildx-cache
Pulumi program:
...
cache_from = "/tmp/.buildx-cache"
cache_to = (
"/tmp/.buildx-cache-new" if os.getenv("GITHUB_ACTIONS") else cache_from
)
image = docker.Image(
"*****",
build=docker.DockerBuild(
context="..",
extra_options=[
"--output=type=docker",
f"--cache-from=type=local,src={cache_from}",
f"--cache-to=type=local,mode=max,dest={cache_to}",
],
),
image_name=repository.repository_url,
registry=registry,
skip_push=False,
)
...
Also, when it comes to the Docker build speed, I wouldn't blame Pulumi. What can take a significant amount of time is the cross-platform build using Buildx (e.g. building for ARM64 on x86-64 GitHub Actions runner). I've experienced this often, mainly with Node.js apps though.bright-orange-69401
09/29/2022, 1:05 PMproud-art-41399
09/29/2022, 1:36 PMmysterious-apartment-62241
10/20/2022, 5:55 PMproud-art-41399
10/20/2022, 6:02 PMbright-orange-69401
10/20/2022, 6:03 PMagreeable-window-77899
11/17/2022, 10:57 AMGitHub Actions cache
? Did you end up building the container separately instead of doing inside the Pulumi code?
Or you are currently using this setup that you mentioned above?proud-art-41399
11/17/2022, 11:36 AM...
- name: Cache Docker layers
uses: actions/cache@v3
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-
...
- name: Move cache
run: |
rm -rf /tmp/.buildx-cache
mv /tmp/.buildx-cache-new /tmp/.buildx-cache
as they are now not needed. And the relevant part of the Pulumi program now looks like this:
...
extra_options = ["--output=type=docker"]
if os.getenv("GITHUB_ACTIONS"):
extra_options.extend(
["--cache-to=type=gha,mode=max", "--cache-from=type=gha"]
)
image = docker.Image(
"xxx",
build=docker.DockerBuild(context="..", extra_options=extra_options),
image_name=repository.repository_url,
registry=registry,
skip_push=False,
)
...
agreeable-window-77899
11/17/2022, 12:02 PMdocker/build-push-action@v3
action instead of the actions/cache@v3
then?proud-art-41399
11/17/2022, 12:03 PMagreeable-window-77899
11/17/2022, 4:16 PMmysterious-apartment-62241
12/23/2022, 8:07 AMtype=gha
and I can see the caches get created automatically!
However, in my case, I am building two Docker Images (via awsx.ecr.buildAndPushImage
), and I don't think the caching is working.
From the detailed diagnostics on Pulumi Cloud, both images seem to be built from scratch, without looking at the cached layers.
After the build, the new image layers are cached.
Do you think there's some additional configuration required for multiple images?qemu
action
⢠the issue seems to occur with / without refresh: true
for the pulumi/push
actionbright-orange-69401
12/23/2022, 9:12 AMawsx.ecr.buildAndPushImage
AFAIK
There's just [awsx.ecr.Image](https://www.pulumi.com/registry/packages/awsx/api-docs/ecr/image/)
If you use the extra_options
like Thomas showed in the code snippet above then you should be fine... ?mysterious-apartment-62241
12/23/2022, 9:30 AMpulumi
and related packages since 3 months ago.
Still, the IaC script works flawlessly. My issue seems to be with the GitHub actions cache. I realized I'm not setting a scope for each image, which is what's probably causing the issue.