https://pulumi.com logo
Title
r

rapid-raincoat-36492

07/06/2022, 2:25 AM
Does using a self-hosted backend affect the concurrency of deploying resources? The stacks I have on app.pulumi.com seem to update all resources in parallel that can be, but my stacks with an S3 backend seem to only update one resource at a time. Is that expected?
b

billowy-army-68599

07/06/2022, 3:36 AM
it largely depends on the backend object store, and is down to the go-cloud underlying library we use to talk to object store buckets
r

rapid-raincoat-36492

07/06/2022, 1:34 PM
Got it, it looks like https://github.com/pulumi/pulumi/issues/4605 has a reference from someone who noticed similar things to me on the S3 backend. If there's any room for me to make a PR to help here, if you could point me to where the S3 communication happens I'll gladly take a look. I know Terraform talks to the S3 backend considerably quicker, and I'm pretty familiar with how they do it in their codebase
b

billowy-army-68599

07/06/2022, 1:59 PM
r

rapid-raincoat-36492

07/06/2022, 3:02 PM
It looks like the blob library exposes
MaxConcurrency
field: https://github.com/google/go-cloud/blob/60d79299da9fb31d6bd022ae834b6766f45b55b1/blob/driver/driver.go#L65 that could be passed through in the
*blob.WriterOptions
of the
WriteAll
method: https://github.com/pulumi/pulumi/blob/ca3b9a57276751c71a4ca7fd10710edcdd72cfd3/pkg/backend/filestate/bucket.go#L22 Would you be open to a change where
pulumi up
could have a new flag named
--max-state-concurrency
where if set, that value would be passed through to the blob for whatever state is used? Looking through the code, it looks like Azure and S3 are the only blob backends that support it right now
b

billowy-army-68599

07/06/2022, 3:17 PM
I'm open to it, but I don't speak for the engineers who manage that code. I'd recommend opening an issue first and discussing there
👍 1
r

rapid-raincoat-36492

07/06/2022, 3:34 PM
Sounds good. Created an issue: https://github.com/pulumi/pulumi/issues/10057
b

billowy-army-68599

07/06/2022, 3:37 PM
Once the thread appears in archive.pulumi.com we can link to it on your issue
r

rapid-raincoat-36492

07/06/2022, 3:38 PM
Oh cool, I'll update the link when I can