https://pulumi.com logo
Title
b

billions-xylophone-85957

08/02/2021, 6:36 PM
what exactly does pulumi CLI upload on
up
? We're using an s3 backend and every change to a rather simple stack (3 stack references, 92 resources) takes 7-8 minutes on 5mbps upload/160ms latency to S3
I understand that this is potato internet, but still
and more importantly, are there any suggestions on reducing the transfer size, whatever that is? The performance of
pulumi up
as far as I can tell is limited by the upload/download speed (nothing could be done about the e.g. cloud providers APIs perf) and local IO to some extent.
b

billowy-army-68599

08/02/2021, 6:51 PM
it's writing your checkpoint file after each API call, ie "what is the current status of the operations on these resources"
b

billions-xylophone-85957

08/02/2021, 6:52 PM
so it's effectively
stack size * number of resources
? Even if the API call is a read?
OK, I did a little log reading and it looks like each "reading" operation is written to the snapshot. Is that intentional?
e.g., a no-op `pulumi --logtostderr --verbose=9 up -y`:
Duration: 4m9s

$ fgrep -c 'Appended new state snapshot to be written' log.txt 
92
$ fgrep -c 'Saved stack' log.txt
13
$ fgrep 'SnapshotManager: mustWrite' log.txt | awk '{print $NF}' | uniq
false
if I'm reading it correctly, the CLI writes the snapshot after processing every resource, even if the resource is
same
all the
mustWrite
checks return false, but there were at least 13 stack writes nonetheless
although it reads like nothing should've been written to the snapshot store:
I0802 20:03:43.882189  389878 snapshot.go:130] SnapshotManager: Beginning mutation for step `same` on resource `urn:foobar`
I0802 20:03:43.882207  389878 step_executor.go:327] StepExecutor worker(0): applying step same on urn:foobar
 (preview false)
I0802 20:03:43.882224  389878 snapshot.go:266] SnapshotManager: sameSnapshotMutation.End(..., true)
I0802 20:03:43.882244  389878 snapshot.go:513] Marked old state snapshot as done: urn:foobar
I0802 20:03:43.882253  389878 snapshot.go:522] Appended new state snapshot to be written: urn:foobar
I0802 20:03:43.882270  389878 snapshot.go:258] SnapshotManager: mustWrite() false
I0802 20:03:43.882280  389878 snapshot.go:288] SnapshotManager: sameSnapshotMutation.End() eliding write
I0802 20:03:43.882290  389878 step_executor.go:327] StepExecutor worker(0): step same on urn:foobar retired
would appreciate any hints to proceed from here
b

billowy-army-68599

08/02/2021, 8:52 PM
yes reading operations being written to the snapshot is intentional, but opening an issue would be the best bet I think
👍 1