Wanted to get some opinions before I file a Github issue/pull-request. The S3 state backend is very slow in my opinion. When compared to Terraform, Pulumi takes much longer to perform updates for 100+ resources using S3 as the backend for the state. I think this is because Pulumi checkpoints the state file to S3 after every action whereas Terraform waits and updates the state after it has finished performing all of its actions. Terraform also locks the state file before doing anything while Pulumi does not, so this might be a blocker to mimicking Terraforms state management behavior. Am I on the right track?

There are a bunch of known issues in this space: https://github.com/pulumi/pulumi/issues/4605, probably https://github.com/pulumi/pulumi/issues/6074 is most applicable from the standpoint of excessive checkpoint related upload traffic. @broad-dog-22463 and @white-balloon-205 might have most context here if you are interested in contributing in this space.

@busy-umbrella-36067 great topics - sorry for missing this thread earlier. I am not super deep on how Terraform’s S3 backend behaves - but I’d be surprised if they don’t save state till the end of an update. That would mean that closing your laptop lid or losing network connection would result in orphaning any resource progress made since the start of the deployment. Regardless - we do not consider that acceptable in the Pulumi model - so we checkpoint progress on each operation. For the Pulumi service backend, the performance overhead of this is nearly always insignificant to the long pole, as these state writes are done in parallel with other cloud provider operations. The original cloud storage support for the filestate backend implementation unfortunately was not particularly well optimized for performance - I don’t recall all the details, but I believe it does several writes (and possibly reads) per storage operation, and I believe because it uses a rename in place operation requires a longer critical-section than the Pulumi service backend that removes some level of parallelism opportunity. The believe there is a lot of room for improvement here - and as @sparse-park-68967 notes, there’s significant investment in improving the filestate backends tracked in 4605. This is definitely an area we’d welcome contribution, and especially if there are opportunities for performance improvements short of overhauling the core filestate backend model. Do let us know if there are particular places you’d like to contribute, and we can find time to discuss in more detail how best to engage in that area.