This command installs plugins for a project https://www.pulumi.com/docs/cli/commands/pulumi_install/ how can I find which plugins I need to install? I'd like to cache plugins for 100+ projects and it looks like that plugin version is downloaded after the query of the state file is pulumi automation SDK a way to go about this or do I need to parallelize

pulumi install

in bash. Ideally I'd like to be able to make a list of required plugins before downloading. Thank you

@great-sunset-355 is this for CI purposes? If so, try to make use of the CI native caching mechanism. For instance, here is the link for Github Actions: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows Pulumi downloads plugins to its home folder. This location can be overridden by setting

PULUMI_HOME

(see docs). Once

PULUMI_HOME

points to such a cache folder, it will fill at first run and Pulumi will pick up matching plugin versions afterwards and no longer download them.

Yes it is for CI purpose. I am aware of

PULUMI_HOME

is not the problem. What I am not sure about is that if Github can incrementally build this cache from multiple jobs. my projects run in parallel and they often have slightly different deps. I am not sure what to use as a cache key. On a small scale, I just ran

pulumi install

for each project in a separate job.

build cache (pulumi install)

-> parallel [

pulumi up <project>

] the problem with this approach that

pulumi install

step may take ages to complete. Ultimately the question is, what to choose as a

cache key

. In my simple monorepo case I went with

package.lock

but if I use this in parallel job it will probably override the cache

Something you could do is have a dedicated job that loops through all your projects to run

pulumi install

, then save cache with a key made up of all lockfiles. The job should run periodically, and whenever the lockfiles change. Then your individual jobs can load the central cache.

It is a monorepo, just one lockfile. Periodic run is not a bad idea though thanks! Parallelized job should only read the cache

They have separate save + restore actions which will help with the implementation: https://github.com/actions/cache/#using-a-combination-of-restore-and-save-actions

thanks, this should get me going! Do you know if the cache restoration will work in another project? I am planning to use a shared workflow and call it from another project.

pulumi-lib=workflow producer

->

app project = workflow consumer

I'm not sure what you mean by project in this case

I have a reusable workflow -> https://docs.github.com/en/actions/using-workflows/reusing-workflows#access-to-reusable-workflows

Cache should work across workflows, however I'm not sure of the behaviour for cross-repo workflows

no problem, I'll do some tests myself