I’ve been thinking about the scopes of Pulumi proj...
# general
I’ve been thinking about the scopes of Pulumi projects 🤔 It’d be interesting if you could have multiple stacks that shared some state. For example, project A that was for a single microservice instance and a project B which was a monolithic project for an entire environment, that included multiple services including project A’s resources. If both shared the same state for this resource, you could apply changes independently. Obviously there’d be a challenge in reading multiple pieces of state across multiple storage backends (eg.GCS buckets); co-ordination/locking; and generally the pitfall of having 2 different (potentially conflicting) declarations of that infrastructure. I then got to thinking about what the scope of a Pulumi project should be. We generally do 1 project per service (whether that’s a business app or a piece of shared infrastructure like RabbitMQ), but I’ve also wanted to co-ordinate lots of these individual projects to spin up an entire environment. I’ve mostly done this via. the automation API but it turns lots of declarative processes into imperative ones, and you lose a lot of the magic of Pulumi. That got me to wondering - does anyone use Pulumi projects in a monolithic approach (ie. encompassing entire environments/clusters)? I can envision a project that deploys the entire production GKE cluster and all of the many apps and infrastructure components. Then you could have a single repo and for the CI/CD pipeline, you could use the
flag to limit the scope of the changes depending upon the trigger (ie. triggered by another repository) . This would be especially easy if you used a
to encapsulate each application’s resource boundarys. Plus you could still get re-use of these individual `ComponentResource`s by packaging them (npm) and importing them. I also think it’d be pretty cool if Pulumi had a first class notion of tags/labels that you could then limit the scope of changes to (eg.
) (Sorry for the brain dump)
It’d be interesting if you could have multiple stacks that shared some state - ie. a project that was for a single microservice instance and a monolithic project that was for an entire environment that included the microservice. Obviously there’d be a challenge in reading multiple pieces of state across multiple storage areas (eg.GCS buckets) and co-ordination/locking.
The popular method for achieving this is with a "shared-" or "base-" project that has the resources which other stacks share. The "child" stacks get the values of the Outputs via StackReferences, and use the resource getters to get a copy of the resource in the current scope. https://www.pulumi.com/blog/iac-recommended-practices-using-stack-references/
I would recommend against the monolithic approach.
interesting - that’s new to me
The more resources you manage, the longer deployments will take just to calculate the resource graph. When your plan fails, it will fail big. Operationally, micro stacks help you limit the damage in the case of an incident.
my gripe is that stacks feel more like different envs (staging/test/prod) instead of different vertical slices
That's where I think you're looking for the Stack References pattern
oh, it works across projects?
I’ll have a read later, thanks
It's also worth noting there is "project level configuration" that all stacks can inherit, but not resources.
Is this relatively new?
Right. Nope.
My organization uses this pattern fairly heavily in conjunction with a few other patterns. 1.
for your account-wide resources 2.
for resources shared across other stacks in a project (e.g. you have a 3 tier web application, and they all have to share some S3 bucket) 3.
for our "level 2" abstractions of our resources (e.g. custom resource) distributed as a library for Stacks to use to simplify resource declarations
do StackReferences work when not using Pulumi’s backend?
Yes, there is a note about a recent change regarding that though
TL;DR, before this change, YOU WILL NOT have "fully qualified stack names" like you do in the Pulumi service. e.g.
The work-around was to create your own namespace convention, and all states live as siblings in the state storage backend. e.g.
One thing to remember about the stack references is that the parent stack can decide to remove the reference and there's no way to notify downstream about the change or removal. So I'd advise combining stack references with lookup aka