Hey folks! We are pursuing a micro-stack architecture but we're bit unclear about how to manage Pulumi stacks in a more dynamic fashion, i.e. how to create/rename/delete them dynamically, including their encryption keys(!), without performing a bunch of manual steps each time. Is there a recommended way to do this? Ideally, I'd just have a piece of Python code that defines a list of stack names, and then Pulumi creates/cleans up stacks automatically.

Have you seen the automation api?

Yes, in fact we already use the Automation API. However, I still think it's cumbersome. Maybe it's because stacks, as they currently implemented, are ultimately the wrong abstraction for a real micro-stack architecture. I would basically like to treat my micro stacks more like any other resource I provision/update/remove.

That sounds more like components than stacks to me. Stacks are necessary book-keeping for cloud state. Theres probably bits we can add to make moving resources easier between them but there's always going to be a level of overhead for them that I'm not sure how we could get rid of.

I would love to use components; however, our infrastructure landscape (and thus our stacks) are gigantic, so we would like to split them up to prevent the entire state from getting locked for a small change in a single component (which hinders parallelization).

I think that would be very very hard with the current backend architecture, but I'll take a look at the ideas in stategraph

Thanks so much! I'd be very interested in hearing your thoughts!

Stategraph sound very interesting, to such a level that I might even test the TF waters when it is out 🙂 I've tried to build a tool that takes all my (pulumi) state-files and merges them into a bigger structure just to be able to query and create diagrams over what is connected to what. This sound like a much more advanced variant of that.

I've tried to build a tool that takes all my (pulumi) state-files and merges them into a bigger structure just to be able to query and create diagrams over what is connected to what.

Good to know I'm not the only one! haha

There is also the (somewhat meta) "pulumi service" provider, which you can use to provision stack resources, and do interesting things like wire up deployment webhooks to cascade updates across dependent stacks automatically.

Interesting, thank you! Unfortunately, it seems it's exclusive to Pulumi Cloud? We use Azure for the storage backend.

ah yeah, PSP is specific for Pulumi Cloud

@echoing-dinner-19531

I think that would be very very hard with the current backend architecture, but I'll take a look at the ideas in stategraph

Have you had a chance to take a look yet?

A bit, but not enough to make any claims or ideas yet

why do you want create and delete stacks dynamically? normally they reflect your environments - production, testing, etc. Just curious what's the use case here?

@high-grass-3103

normally they reflect your environments - production, testing, etc.

Yes, they normally do if you're developing & deploying a single application. However, we are the platform team of our organization and administer a whole landscape of infrastructure for our application teams. For instance, every application team gets a landing zone from us. Now, we could try to deploy all landing zones within a single Pulumi stack (i.e. basically a

for

loop over the list of application teams). However, we did that in the past and ended up with pipelines that would take hours to finish. Moreover, during that time the Pulumi state would be locked, so you could not deploy any other changes in parallel. (E.g. deploy changes to landing zones 1 and 2 while that other pipeline is still working on landing zones 99 and 100.) This is why we need more fine-grained locking. The best approach would probably be the one stategraph.dev is taking. However, right now in Pulumi a micro-stack architecture, in which every landing zone amounts to a separate stack, seems to be the only viable option. But then you're basically looking at dynamically creating & deleting stacks like any other resource. And managing cross-stack dependencies. Note that landing zones are but one of many examples where micro-stacks would be useful for us. The same situation arises, e.g., in our Azure API Management instance (basically an API gateway for our organization), where we would like to separate the deployments of individual APIs from each other because they are rather error prone. (Azure applies validation rules that are not entirely obvious.) I could keep going…