Hi all! I've recently began using Pulumi's automa...
# general
l
Hi all! I've recently began using Pulumi's automation API to manage Cloudfront Distributions and ran into an efficiency problem. Wondering if someone here could help me out: • I commonly use 4+ distributions for my environments. • Each distribution is taking about 3 minutes to complete. • They all appear to be deploying serially, i.e. one after another. • So, about 12 minutes per execution of the deployment script. _The actual question_: Is it possible to deployment the distributions in parallel instead of one after another? edit: Sorry for all the edits. My brain is frazzled after coding all day.
1
e
Is this 4+ distributions in one pulumi program? I.e. one call to
up
And is there any chance there's data dependencies between the distributions? Your not using outputs of one to set inputs of another?
Because if both of the above are the case I'd expect the engine to already issue these in parallel.
l
They're all being deployed in the same call to
up
, but they're created in a loop. Your instict makes me think to me that a call to
aws.cloudfront.get_distributions
in the same loop may be causing it to block. Going to shuffle that logic a bit. Thanks for the hint!
e
What language are you using?
I wouldn't expect get_distributions to block, except in Go because no tasks/promises there.
l
Python. Here's a scap of the PoC source.
Destroying the stack definitely happens in parallel.
e
Thanks it's getting a bit late to go code digging here now, but I'll make a note to check this in the morning.
l
Woot! Seems that
get_distribution
was blocking. I thought I was calling it such that it'd return an output object but I must have been calling it wrong. This is working in parallel. Really appreciate your tip!
e
Good to hear, I'll see if I can work out why its blocking because that's surprising to me and would be good to fix.
👍 1
l
Makes total sense. I'm still novice at Pulumi, but let me know if there's anything I can do to assist. Really loving it tho. Revolutionary product IMHO.
❤️ 1
🙏 1
e
Looks like invokes in python are blocking, odd that they still return an awaitable. Something for us to improve in a future version.
l
Ah, alright. Really wish I was more familiar with the project's architecture so I could help improve it. At least it seems we have a scent to track now!