Hello everyone ! We've been using Pulumi for a couple years now, and we just ended up seeing an issue: backup and history files number. Each time we launch a

pulumi up

, and in our design it could be each minute or so, and we've just seen our storage bloated by backup files, as well as history files. We're managing ~4000 stacks, and some have more than 300 000 files. Most of these files are backups and history. Some stack are > 200GB. It seems that there are no way to limit, nor purge backup files directly with pulumi CLI. For that, I've created a little algorithm to keep only the latest. Safe to use (even if limiting these backup files with pulumi CLI would be better :D). It seems also that there are no way to limit checkpoint and history files. But this ones feel trickier. I guess we can't just remove the oldest ones ? As if I understand it well, it's incremental files. And I wonder what would happen if I end up removing oldest files, like for backups ? Is there any guidance so far to keep those files under a reasonable rate, and is there any "danger" behind removing history and checkpoint files ?

From my perspective the most important file is state. The rest is more useful for pulumi itself or if you need history of changes that you applied. In my project I use a set of environment variables to limit amount (skip, disable, retain) and size of files (gzip) that were generated

export PULUMI_EXPERIMENTAL="true"
export PULUMI_SKIP_CHECKPOINTS="true"
export PULUMI_SELF_MANAGED_STATE_GZIP="true"
export PULUMI_DISABLE_CHECKPOINT_BACKUPS="true"
export PULUMI_RETAIN_CHECKPOINTS="false"

Thanks, never thought of envs to manage that. I'll give it a try

I don't think so. I spent quite some time reading blog posts, Github issues and source code to complete that list of environment variables