Is there any way to speed up the transfer of s3 objects I ha

Question

Is there any way to speed up the transfer of s3 objects I have a directory of images that is taking a very long time to upload via pulumi but only takes a few seconds via `aws s3 sync` I tried doing a sync and refresh first so pulumi could just handle deltas but it wound up recreating all the files the first time I ran `up` so it didn t save me any time edited

billowy-army-68599 · Answer

hmm how long is it taking

tall-nightfall-10556 · Answer

it s about 1sec per file

tall-nightfall-10556 · Answer

so for a directory with ~500 files it s about 10 minutes

tall-nightfall-10556 · Answer

vs 2 3 sec with aws sync

billowy-army-68599 · Answer

i think it s because of the design `aws sync` is just a splat whereas for each object in Pulumi we have to upload store the result of that update in the state checkpoint which is going to have an impact

billowy-army-68599 · Answer

some users have switched to a dynamic provider to manage objects instead

tall-nightfall-10556 · Answer

is there an example of the dynamic provider approach

billowy-army-68599 · Answer

billowy-army-68599 · Answer

knew you were gonna ask so went looking smile

tall-nightfall-10556 · Answer

Or what I was hoping was to seed the state with sync and then do an import of the objects with refresh so it could handle deltas from that point

billowy-army-68599 · Answer

the caveat here is that it won t manage the objects on delete it has no delete in the crud

billowy-army-68599 · Answer

i m not sure if you could seed the state with sync you d have to import every object into the state which is going to take just as long I suspect disappointed