what exactly does pulumi CLI upload on `up`? We're...
# general
b
what exactly does pulumi CLI upload on
up
? We're using an s3 backend and every change to a rather simple stack (3 stack references, 92 resources) takes 7-8 minutes on 5mbps upload/160ms latency to S3
I understand that this is potato internet, but still
and more importantly, are there any suggestions on reducing the transfer size, whatever that is? The performance of
pulumi up
as far as I can tell is limited by the upload/download speed (nothing could be done about the e.g. cloud providers APIs perf) and local IO to some extent.
b
it's writing your checkpoint file after each API call, ie "what is the current status of the operations on these resources"
b
so it's effectively
stack size * number of resources
? Even if the API call is a read?
OK, I did a little log reading and it looks like each "reading" operation is written to the snapshot. Is that intentional?
e.g., a no-op `pulumi --logtostderr --verbose=9 up -y`:
Copy code
Duration: 4m9s

$ fgrep -c 'Appended new state snapshot to be written' log.txt 
92
$ fgrep -c 'Saved stack' log.txt
13
$ fgrep 'SnapshotManager: mustWrite' log.txt | awk '{print $NF}' | uniq
false
if I'm reading it correctly, the CLI writes the snapshot after processing every resource, even if the resource is
same
all the
mustWrite
checks return false, but there were at least 13 stack writes nonetheless
although it reads like nothing should've been written to the snapshot store:
Copy code
I0802 20:03:43.882189  389878 snapshot.go:130] SnapshotManager: Beginning mutation for step `same` on resource `urn:foobar`
I0802 20:03:43.882207  389878 step_executor.go:327] StepExecutor worker(0): applying step same on urn:foobar
 (preview false)
I0802 20:03:43.882224  389878 snapshot.go:266] SnapshotManager: sameSnapshotMutation.End(..., true)
I0802 20:03:43.882244  389878 snapshot.go:513] Marked old state snapshot as done: urn:foobar
I0802 20:03:43.882253  389878 snapshot.go:522] Appended new state snapshot to be written: urn:foobar
I0802 20:03:43.882270  389878 snapshot.go:258] SnapshotManager: mustWrite() false
I0802 20:03:43.882280  389878 snapshot.go:288] SnapshotManager: sameSnapshotMutation.End() eliding write
I0802 20:03:43.882290  389878 step_executor.go:327] StepExecutor worker(0): step same on urn:foobar retired
would appreciate any hints to proceed from here
b
yes reading operations being written to the snapshot is intentional, but opening an issue would be the best bet I think
👍 1