alright 👏 just finished migrating my monolith to microstacks AMA 🪩 some thoughts... • localized microstack deployments are much faster, saving a lot of time that was previously wasted on unnecessary chatter with cloud providers • having the microstack colocated with its app or service is a massive win, especially as the codebase grows • the DX to support a microstack migration is a painful omission... i.e. a

pulumi transfer

command for migrating one or more resources would be super handy (tracked in GH issue 3389) • you can use

pulumi import

or the

import

resource option, but again the DX here is not built for a microstack migration and is quite problematic (e.g "inputs to import do not match the existing resource; importing this resource will fail") • imports are also dependant on the provider implementation. this is how I discovered the upstash provider importers weren't working... 🔥

transfer

proposal would bypass provider importers • ... in the end I just abandoned the old resources and created new resources in the microstacks + performed manual data migrations • it couples really well with nx! each

pulumi up

command is wrapped in an nx task, with dependencies modelled using the

dependsOn

option so that when bringing up a new environment it knows which order to deploy the microstacks. this also ensures the stack references won't fail • related to this - if i want to create a new environment for just a single microservice, I can run the nx task for that deployment and it will also deploy only the subject & dependencies into the new environment, which is brilliant. might even look into stub microstacks later on down the track if they can help with spinning up new sparse environments for focused work/experimentation/testing • nx distributed task caching & "affected projects" calculations also work really well to avoid doing any work that isn't needed, including skipping the

pulumi up

command entirely if input assets or the pulumi program code are unchanged • I've also got the notion of 2 types of stack programs - cloud and local, with the idea that local can be run completely offline e.g. on a long flight / train ride. For some microstacks it was as simple as a few

isLocal ? x : y

ternaries but for others I needed to use completely seperate programs. All you need to do is ensure the outputs match, and then use a helper like

getStackRef('service-b')

that returns the matching environment stack for dependencies.

Yeah, ESC requires Pulumi Cloud. Thanks for all this feedback; I’ve passed it on to our internal teams.

Awesome feedback - thanks @icy-controller-6092! A few notes/questions.

localized microstack deployments are much faster, saving a lot of time that was previously wasted on unnecessary chatter with cloud providers

I believe you are using the filestate backend (S3 or local)? I would be a bit surprised if there was any material unnecessary chatter with cloud providers outside of state storage - in general we don't have to speak to the cloud providers except when there is an actual CRUD operation to perform on a resource that is changing. And in the Pulumi Cloud state backend, we have some optimizations based on compute on the server to limit the chatter to diffs, which should ensure these costs don't scale with stack size. In general, it is a goal for us to ensure that effectively no performance costs scale with stack size - so would love to undersand whether these costs for you are coming from state store or from resource CRUD operations! (Aside - there are of course other very good reasons to break up large stacks, such as for reduced blast radius, seperate versioning, etc.).

the DX to support a microstack migration is a painful omission... i.e. a

pulumi transfer

command for migrating one or more resources would be super handy (tracked in GH issue 3389)

Absolutely - this is something we really want to add to the CLI! As you note - this is tracked in https://github.com/pulumi/pulumi/issues/3389, which is currently the 8th most requested feature in the entire Pulumi open source project, and one we hope to tackle very soon.

it couples really well with nx! each

pulumi up

command is wrapped in an nx task, with dependencies modelled using the

dependsOn

option so that when bringing up a new environment it knows which order to deploy the microstacks. this also ensures the stack references won't fail

This is great to hear. We've seen many users moving to NX recently, and have work on improved support for NX that we're expecting to tackle this quarter in https://github.com/pulumi/pulumi/issues/8865.

I've also got the notion of 2 types of stack programs - cloud and local, with the idea that local can be run completely offline

Curious - what do you have in your

local

stacks that doesn't require talking to the cloud? Is this local Kubernetes cluster and/or workload resources? Something else?

to get around this i wrote a shell script to grab the outputs from the "infra" microstack and then add them to each app microstack using

pulumi conig set ...

someone mentioned you can use ESC for this, but for my local stacks im using a local pulumi backend (

pulumi login file:///...

) and ESC won't work then

Yeah - this is a great use case for ESC. It offer the ability to pull configuration out of Pulumi Stack outputs, and then to project it into configuration for a stack. Avoids a ton of copy/paste!

Isn't it awesome when the CTO of the company takes the time to respond to your feedback? 🤩 Thanks Luke!

It's very awesome, great to hear from you Luke 🙂 I think the first part that I hadn't explained well is Cloud vs Local - these can be thought of as templates. The cloud stacks (e.g. prod, staging) use the Pulumi Cloud backend, the local stacks (e.g. local1, local2) use a local file backend on the dev laptop - the requirement is that the local cluster should be deployable to a local docker/k8s cluster, even when the developer is completely offline. The Local-type stacks replace the previous docker-compose.yaml implementation for local dev - which was very brittle, required a lot of hand-holding and wiring values via env vars was error prone. Now both stack types use the same wiring logic via stack references. In fact, after getting the Local-type stack working first and then deploying the Cloud-type stack, there were zero integration issues 🎆 due to Local and Cloud now both using the same wiring mechanics. An example of where the microstacks implementation differs might be (1) cloud deploys confluence, local deploys redpanda; or (2) cloud uses upstash, local uses a local redis image. The important thing here is that the outputs of a microstack remain the same for both a cloud and a local type microstack, e.g. an output of an infra-level microstack could be named

redis_conn

and the dependants will use a StackReference to that output, and are completely agnostic to whether the string is an Upstash or local docker image connection. This means that I can't use ESC to configure default providers, because even if I did use ESC for the Cloud-type stacks I would still require a solution for Local-type stacks and I didn't want to manage that branching, so I just a shell script to copy provider config outputs into the relevant stack yaml files > in general we don't have to speak to the cloud providers except when there is an actual CRUD operation to perform on a resource that is changing I use the

--refresh

flag when running

pulumi up

so the more resources, the more API calls to AWS, Databricks, Upstash, etc. Even without a pre-up refresh the microstack programs still feel much snappier, I suspect this is due to there being a much lower resource count in a microstack versus the number of resources in a monolith stack, plus the benefits you mention are also a huge bonus

@icy-controller-6092 Thanks for this excellent write up! Can i pick your brain as I am trying to figure out a deployment process for my microstacks that will work well. I saw you mention "having the microstack colocated with its app or service" and also using nx. My question to you is do you have a seperate monorepo for managing your infra project using nx or if you have your pulumi code in the same monorepo that also contains your application code? Also curious to if you have done any CI/CD with this configuration and any lessons learned.