Is there any way to speed up the transfer of s3 ob...
# general
t
Is there any way to speed up the transfer of s3 objects?  I have a directory of images that is taking a very long time to upload via pulumi, but only takes a few seconds via 
aws s3 sync
. I tried doing a sync and refresh first so pulumi could just handle deltas, but it wound up recreating all the files the first time I ran 
up
, so it didn't save me any time. (edited)
b
hmm, how long is it taking?
t
it's about 1sec per file.
so for a directory with ~500 files, it's about 10 minutes
vs 2-3 sec with aws sync
b
i think it's because of the design.
aws sync
is just a splat, whereas for each object in Pulumi we have to upload, store the result of that update in the state/checkpoint, which is going to have an impact
some users have switched to a dynamic provider to manage objects instead
t
is there an example of the dynamic provider approach?
knew you were gonna ask so went looking 😄
t
Or what I was hoping was to seed the state with sync and then do an import of the objects with refresh so it could handle deltas from that point.
b
the caveat here is that it won't manage the objects on delete, it has no delete in the crud
i'm not sure if you could seed the state with sync, you'd have to import every object into the state which is going to take just as long I suspect 😞