Hi all, I have a bit of a wild issue (at least to me) and not found any kind of resource about it yet.
We are using Pulumi to manage the state of our 3 environments (dev, staging, prod) and the staging envrionment is always slow. Slow meaning 4+ minutes for previews and updates while dev and prod are done in 20ish seconds or faster.
It doesn't matter which operation I am running, it doesn't matter which resource is updated (or no resource at all).
I have enabled debug logs and compared a pulumi preview
on prod vs staging (with no changes in each of them) and found things like
• ~On staging it's doing LOTS of
<https://api.pulumi.com/api/stacks/[COMPANY> NAME]/[COMPANY NAME]/staging/decrypt
calls at the start, specifically there's 2 blocks of lots of these calls which roughly add
2 minutes each (so this is the issue)
•
Staging has 50 resources less than prod but is slower
•
The log files on staging is 20MB smaller (just interesting I thought)
Does anyone have an idea on what this could be / how to debug this more? It's starting to drive us nuts
I am an absolute idiot, for some reason staging didnt have our own Azure KV as secrets prvider while the other stacks had it. I'll leave this here because others may be as stupid and find this but god damn this is stupid