https://pulumi.com logo
#general
Title
# general
g

great-sunset-355

12/06/2023, 1:52 PM
This command installs plugins for a project https://www.pulumi.com/docs/cli/commands/pulumi_install/ how can I find which plugins I need to install? I'd like to cache plugins for 100+ projects and it looks like that plugin version is downloaded after the query of the state file is pulumi automation SDK a way to go about this or do I need to parallelize
pulumi install
in bash. Ideally I'd like to be able to make a list of required plugins before downloading. Thank you
l

limited-rainbow-51650

12/06/2023, 3:46 PM
@great-sunset-355 is this for CI purposes? If so, try to make use of the CI native caching mechanism. For instance, here is the link for Github Actions: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows Pulumi downloads plugins to its home folder. This location can be overridden by setting
PULUMI_HOME
(see docs). Once
PULUMI_HOME
points to such a cache folder, it will fill at first run and Pulumi will pick up matching plugin versions afterwards and no longer download them.
g

great-sunset-355

12/06/2023, 4:54 PM
Yes it is for CI purpose. I am aware of
PULUMI_HOME
is not the problem. What I am not sure about is that if Github can incrementally build this cache from multiple jobs. my projects run in parallel and they often have slightly different deps. I am not sure what to use as a cache key. On a small scale, I just ran
pulumi install
for each project in a separate job.
build cache (pulumi install)
-> parallel [
pulumi up <project>
] the problem with this approach that
pulumi install
step may take ages to complete. Ultimately the question is, what to choose as a
cache key
. In my simple monorepo case I went with
package.lock
but if I use this in parallel job it will probably override the cache
d

dry-keyboard-94795

12/06/2023, 5:17 PM
Something you could do is have a dedicated job that loops through all your projects to run
pulumi install
, then save cache with a key made up of all lockfiles. The job should run periodically, and whenever the lockfiles change. Then your individual jobs can load the central cache.
Incremental cache for github actions isn't native functionality. You can do it by loading the cache, running the program to load new files, purging the cache server side (using Github apis), then saving the cache. The cache key should be all lockfiles from projects partaking in the caching. With jobs running in parallel, I'd be concerned about race conditions where a purge happens, and another job ends up not loading a cache. The provided locking mechanisms in GH are optimistic instead of guaranteed, so this approach will need your own external locking system too.
As it's one of your concerns, ghactions won't override cache. Once it's saved, it can't be written to again.
g

great-sunset-355

12/06/2023, 7:15 PM
It is a monorepo, just one lockfile. Periodic run is not a bad idea though thanks! Parallelized job should only read the cache
d

dry-keyboard-94795

12/06/2023, 7:21 PM
They have separate save + restore actions which will help with the implementation: https://github.com/actions/cache/#using-a-combination-of-restore-and-save-actions
g

great-sunset-355

12/07/2023, 5:40 AM
thanks, this should get me going! Do you know if the cache restoration will work in another project? I am planning to use a shared workflow and call it from another project.
pulumi-lib=workflow producer
->
app project = workflow consumer
d

dry-keyboard-94795

12/07/2023, 7:28 AM
I'm not sure what you mean by project in this case
g

great-sunset-355

12/07/2023, 7:36 AM
d

dry-keyboard-94795

12/07/2023, 7:37 AM
Cache should work across workflows, however I'm not sure of the behaviour for cross-repo workflows
g

great-sunset-355

12/07/2023, 7:45 AM
no problem, I'll do some tests myself