glamorous-printer-66548
10/29/2018, 11:08 PMcacheFrom
almost impossible. In order to effectively use cacheFrom
I need some stable tag (i.e. latest
) or some other identifier that can be derived from logic in the code before the build runs. Prior to your change I was adding latest
as tag to all images which in conjunction with cacheFrom: true
enabled quite good caching. Any thoughts how I can achieve good caching now?lemon-spoon-91807
10/29/2018, 11:10 PMglamorous-printer-66548
10/29/2018, 11:11 PMlatest
- <git_sha>
and then we pushed both tags.
In our k8s deployment we would reference the docker image with the <git_sha>
tag but during docker build we would pull and --cache-from=<image_name>:latestlatest
that somewhat solves my problem.lemon-spoon-91807
10/29/2018, 11:15 PMcacheFrom: {stages: ["some_id"]}
glamorous-printer-66548
10/29/2018, 11:18 PMlemon-spoon-91807
10/29/2018, 11:18 PMglamorous-printer-66548
10/29/2018, 11:21 PMstages
are named stages in a multi-stage Dockerfile. A multi-stage dockerfile contains multiple FROM
clauses.. i.e. check thiose docs: https://docs.docker.com/develop/develop-images/multistage-build/#use-multi-stage-builds
.
If I would specify latest
as stage name, pulumi’s docker code would attempt to build the dockerfile with docker build . --target latest
which would fail unless the dockerfile contains a named latest
stage.
Or to cut a long story short: stages have nothing to do with tags.lemon-spoon-91807
10/29/2018, 11:22 PMglamorous-printer-66548
10/29/2018, 11:24 PMlatest
.<image_id>
or latest-<image_id>
. So basically the tag gets unstable which makes it unsuitable for caching.lemon-spoon-91807
10/29/2018, 11:26 PMglamorous-printer-66548
10/29/2018, 11:26 PMlemon-spoon-91807
10/29/2018, 11:26 PMglamorous-printer-66548
10/29/2018, 11:26 PMlemon-spoon-91807
10/29/2018, 11:27 PMAnd everytime I make a code change now it will build the entire image completely from scratch.
glamorous-printer-66548
10/29/2018, 11:27 PMlatest
and using the latest
tag as --cache-from is a very simple strategy that speeds up builds in many cases, but a slightly more sophisticated and better strategy would be probably this:
1. tag and push each image with the git_sha of HEAD
2. before build pulumi should attempt to pull <image_name>:<git_sha_HEAD>
. If it could successfully pull this, use it --cache-from . If not, attempt to pull <image_name>:<git_sha_HEAD~1>
and use this as --cache from. Repeat this process until an image could be pulled successfully (maybe stop doing so after 5 iterations or until HEAD~5).lemon-spoon-91807
10/29/2018, 11:39 PMglamorous-printer-66548
10/29/2018, 11:47 PMlemon-spoon-91807
10/30/2018, 12:13 AMIf not, attempt to pull <image_name>:<git_sha_HEAD~1>
This needs probably some additional thought for determining what tag to push if the git working directory was dirty at the time of build.
glamorous-printer-66548
10/30/2018, 12:23 AMwouldn’t that discount any changes you made yourself?Nope, if I use
<git_sha_HEAD~1
as cache-source docker will attempt to reuse as many layers as possible from the cached image, but it will still detect local source code changes or local dockerfile changes (that are different from the cached image) and build the corresponding layers from scratch.lemon-spoon-91807
10/30/2018, 12:24 AMglamorous-printer-66548
10/30/2018, 12:32 AMcacheFrom: {stages: ...}
I would have to add multiple stages to each dockerfile which is some additional work (and shouldn’t be required to get good caching). Second I think after your change even when using cacheFrom: {stages:... }
each stage will be tagged with something like <stageName>-<image_id>
. So each of the stages doesn’t have a predictable tag to pull from either.lemon-spoon-91807
10/30/2018, 12:33 AMglamorous-printer-66548
10/30/2018, 12:34 AMlemon-spoon-91807
10/30/2018, 12:38 AMglamorous-printer-66548
10/30/2018, 12:42 AM