https://pulumi.com logo
Title
l

lemon-church-28946

07/13/2022, 7:28 PM
Hi all! I've recently began using Pulumi's automation API to manage Cloudfront Distributions and ran into an efficiency problem. Wondering if someone here could help me out: • I commonly use 4+ distributions for my environments. • Each distribution is taking about 3 minutes to complete. • They all appear to be deploying serially, i.e. one after another. • So, about 12 minutes per execution of the deployment script. _The actual question_: Is it possible to deployment the distributions in parallel instead of one after another? edit: Sorry for all the edits. My brain is frazzled after coding all day.
1
e

echoing-dinner-19531

07/13/2022, 7:47 PM
Is this 4+ distributions in one pulumi program? I.e. one call to
up
And is there any chance there's data dependencies between the distributions? Your not using outputs of one to set inputs of another?
Because if both of the above are the case I'd expect the engine to already issue these in parallel.
l

lemon-church-28946

07/13/2022, 7:51 PM
They're all being deployed in the same call to
up
, but they're created in a loop. Your instict makes me think to me that a call to
aws.cloudfront.get_distributions
in the same loop may be causing it to block. Going to shuffle that logic a bit. Thanks for the hint!
e

echoing-dinner-19531

07/13/2022, 7:54 PM
What language are you using?
I wouldn't expect get_distributions to block, except in Go because no tasks/promises there.
l

lemon-church-28946

07/13/2022, 7:57 PM
Python. Here's a scap of the PoC source.
Destroying the stack definitely happens in parallel.
e

echoing-dinner-19531

07/13/2022, 8:00 PM
Thanks it's getting a bit late to go code digging here now, but I'll make a note to check this in the morning.
l

lemon-church-28946

07/13/2022, 8:01 PM
Woot! Seems that
get_distribution
was blocking. I thought I was calling it such that it'd return an output object but I must have been calling it wrong. This is working in parallel. Really appreciate your tip!
e

echoing-dinner-19531

07/13/2022, 8:02 PM
Good to hear, I'll see if I can work out why its blocking because that's surprising to me and would be good to fix.
👍 1
l

lemon-church-28946

07/13/2022, 8:06 PM
Makes total sense. I'm still novice at Pulumi, but let me know if there's anything I can do to assist. Really loving it tho. Revolutionary product IMHO.
❤️ 1
:thank-you: 1
e

echoing-dinner-19531

07/14/2022, 8:53 AM
Looks like invokes in python are blocking, odd that they still return an awaitable. Something for us to improve in a future version.
l

lemon-church-28946

07/14/2022, 2:08 PM
Ah, alright. Really wish I was more familiar with the project's architecture so I could help improve it. At least it seems we have a scent to track now!