https://pulumi.com logo
#general
Title
# general
t

tall-nightfall-10556

10/06/2021, 5:36 PM
Is there any way to speed up the transfer of s3 objects?  I have a directory of images that is taking a very long time to upload via pulumi, but only takes a few seconds via 
aws s3 sync
. I tried doing a sync and refresh first so pulumi could just handle deltas, but it wound up recreating all the files the first time I ran 
up
, so it didn't save me any time. (edited)
b

billowy-army-68599

10/06/2021, 5:37 PM
hmm, how long is it taking?
t

tall-nightfall-10556

10/06/2021, 5:38 PM
it's about 1sec per file.
so for a directory with ~500 files, it's about 10 minutes
vs 2-3 sec with aws sync
b

billowy-army-68599

10/06/2021, 5:40 PM
i think it's because of the design.
aws sync
is just a splat, whereas for each object in Pulumi we have to upload, store the result of that update in the state/checkpoint, which is going to have an impact
some users have switched to a dynamic provider to manage objects instead
t

tall-nightfall-10556

10/06/2021, 5:41 PM
is there an example of the dynamic provider approach?
knew you were gonna ask so went looking 😄
t

tall-nightfall-10556

10/06/2021, 5:41 PM
Or what I was hoping was to seed the state with sync and then do an import of the objects with refresh so it could handle deltas from that point.
b

billowy-army-68599

10/06/2021, 5:42 PM
the caveat here is that it won't manage the objects on delete, it has no delete in the crud
i'm not sure if you could seed the state with sync, you'd have to import every object into the state which is going to take just as long I suspect 😞