https://pulumi.com logo
#general
Title
# general
b

bland-spring-92723

02/28/2024, 8:43 AM
Hello everyone ! We've been using Pulumi for a couple years now, and we just ended up seeing an issue: backup and history files number. Each time we launch a
pulumi up
, and in our design it could be each minute or so, and we've just seen our storage bloated by backup files, as well as history files. We're managing ~4000 stacks, and some have more than 300 000 files. Most of these files are backups and history. Some stack are > 200GB. It seems that there are no way to limit, nor purge backup files directly with pulumi CLI. For that, I've created a little algorithm to keep only the latest. Safe to use (even if limiting these backup files with pulumi CLI would be better :D). It seems also that there are no way to limit checkpoint and history files. But this ones feel trickier. I guess we can't just remove the oldest ones ? As if I understand it well, it's incremental files. And I wonder what would happen if I end up removing oldest files, like for backups ? Is there any guidance so far to keep those files under a reasonable rate, and is there any "danger" behind removing history and checkpoint files ?
g

great-zebra-31498

02/28/2024, 11:07 AM
From my perspective the most important file is state. The rest is more useful for pulumi itself or if you need history of changes that you applied. In my project I use a set of environment variables to limit amount (skip, disable, retain) and size of files (gzip) that were generated
Copy code
export PULUMI_EXPERIMENTAL="true"
export PULUMI_SKIP_CHECKPOINTS="true"
export PULUMI_SELF_MANAGED_STATE_GZIP="true"
export PULUMI_DISABLE_CHECKPOINT_BACKUPS="true"
export PULUMI_RETAIN_CHECKPOINTS="false"
b

bland-spring-92723

02/28/2024, 1:05 PM
Thanks, never thought of envs to manage that. I'll give it a try
is there any updated documentation about all the environment variables possible ? This one is incomplete: https://www.pulumi.com/docs/cli/environment-variables/
g

great-zebra-31498

02/28/2024, 7:41 PM
I don't think so. I spent quite some time reading blog posts, Github issues and source code to complete that list of environment variables