https://pulumi.com logo
Title
b

bright-orange-69401

04/19/2023, 8:21 AM
I’m having tons of problems with
pulumi-docker
4.x For one, it seems that Pulumi team created a hashing function that hashes all the Docker context (minus the .dockerignore) before actually running the build with docker’s engine : is there a way to disable this and instead let Docker build the image and compute the actual digest ? (source: https://github.com/pulumi/pulumi-docker/blob/1966f086164d1e1b87d96c9a9da81efef1167816/provider/provider.go#L162) In our case, we have a structure like this:
projects/
libs/
So we usually set the Docker context at
.
and use multi-stage builds to build our different projects : this way we can share the libs that are built at earlier stages in some of the projects Although the new docker provider does support the
target
argument, the
hashContext
function does not care for multi-stage builds : it calculates a single hash for the whole context... no matter which stage we actually intend to build The other -much more problematic- issue we’re having is that altering a file (such as adding a comment
# foo
somewhere) does not trigger an update when running
pulumi up
I can’t know for sure what’s happening, but I know for a fact that running
docker build
manually does pick up the change and update the digest of the image, whereas pulumi-docker’s pre-hash seems to block the docker engine from actually running This is preventing us from moving forward on our prod releases 😞 @limited-rainbow-51650 is there anybody at Pulumi that could chime in on this ?
s

salmon-account-74572

04/19/2023, 4:46 PM
Hi Louis! I’ve pinged some of the internal team for the Docker provider with a link to this post. If they don’t respond here directly, I’ll relay the feedback. Thanks for sharing your concerns! It might also be good to go ahead and open an issue (if you haven’t already) with all these details as well (if you don’t mind).
o

orange-policeman-59119

04/19/2023, 5:10 PM
Hey @bright-orange-69401 - can you confirm that you're on the latest version of the provider? We've made some very specific fixes to context hashing, other steps. Regarding build stages, I'm not sure what you mean? All build stages utilize the same context, I've confirmed this by comparing our provider to the Docker CLI's implementation. We now support caching multi-stage builds via
cacheFrom
, and setting the build arg
BUILDKIT_INLINE_CACHE=1
so I don't think you need to build specific stages any longer to obtain caching benefits. There is a caveat: there is an outstanding bug in Docker that causes inline caching to fail every other build, but we may have a workaround in a coming release.
b

bright-orange-69401

04/19/2023, 5:12 PM
Hey @orange-policeman-59119
can you confirm that you’re on the latest version of the provider?
Yes, as of this morning at least
o

orange-policeman-59119

04/19/2023, 5:14 PM
I'd be happy to hop on a call to talk through the changes in v4 and create issue(s) tracking anything we think is a defect. Computing the build context hash ourselves is an integral part of the provider, though - it is necessary in order for us to detect the correct operation that will be performed in
preview
.
b

bright-orange-69401

04/19/2023, 5:14 PM
Regarding build stages, I’m not sure what you mean? All build stages utilize the same context, I’ve confirmed this by comparing our provider to the Docker CLI’s implementation.
That’s my point : when you build a
foo
stage, your image digest is not gonna be the same as a
bar
stage on the same Dockerfile, even though they have the same context So the
hashContext
function that calculates the context’s hash and then decides not to actually run the docker engine is actually causing problems for us
I’d be happy to hop on a call to talk through the changes in v4 and create issue(s) tracking anything we think is a defect.
Sure, I’m available if you are
b

bright-orange-69401

04/19/2023, 6:47 PM
Hello again @orange-policeman-59119 @shy-arm-32391 I’ve managed to do a minimal repro here using pure YAML: https://github.com/LouisAmon/pulumi-docker-4-bug There’s 100% something fishy going on : the bug seems to happen depending on the level of complexity of the directory structure as illustrated in the repro
s

shy-arm-32391

04/20/2023, 11:01 PM
Thank you for this repro - it was so helpful and user friendly! :meow-party: I am filing an issue for this and I have a potential fix in the works.