Hello everyone ! We've been using Pulumi for a cou...
# general
Hello everyone ! We've been using Pulumi for a couple years now, and we just ended up seeing an issue: backup and history files number. Each time we launch a
pulumi up
, and in our design it could be each minute or so, and we've just seen our storage bloated by backup files, as well as history files. We're managing ~4000 stacks, and some have more than 300 000 files. Most of these files are backups and history. Some stack are > 200GB. It seems that there are no way to limit, nor purge backup files directly with pulumi CLI. For that, I've created a little algorithm to keep only the latest. Safe to use (even if limiting these backup files with pulumi CLI would be better :D). It seems also that there are no way to limit checkpoint and history files. But this ones feel trickier. I guess we can't just remove the oldest ones ? As if I understand it well, it's incremental files. And I wonder what would happen if I end up removing oldest files, like for backups ? Is there any guidance so far to keep those files under a reasonable rate, and is there any "danger" behind removing history and checkpoint files ?
From my perspective the most important file is state. The rest is more useful for pulumi itself or if you need history of changes that you applied. In my project I use a set of environment variables to limit amount (skip, disable, retain) and size of files (gzip) that were generated
Copy code
Thanks, never thought of envs to manage that. I'll give it a try
is there any updated documentation about all the environment variables possible ? This one is incomplete: https://www.pulumi.com/docs/cli/environment-variables/
I don't think so. I spent quite some time reading blog posts, Github issues and source code to complete that list of environment variables