Is there a perspective of best practice around structuring projects and stacks? My specific question is around something like this: we have an

infra

project that provisions all the infrastructure needed (e.g. EKS, RDS, VPC and so on), and then a

tenant

project where we provision some per-tenant resources in an individual stack per tenant (and this uses a stack reference back to an

infra

- which one to use in the stack config). So now, if I want to have dev/staging/prod type separation, I can do it in a couple of high level ways: 1. One project for

infra

and then just name them appropriately or set some property on each stack which type of environment it lives in. 2. A project for

infra

per environment (dev/staging/prod). Same options exist for the

tenant

project as well (i.e. one project for all envs or one per env). Code duplication is not a huge issue - 99% of the code is in some shared libraries, so the

main.go

for us is very small. The main tradeoff in my mind is that with (1), we get the simplicity of a single project, but now if we want to do something for “all staging stacks”, it becomes a much more complex operation (need to decide how to distinguish them, etc). For (2), it’s kind of reversed. I was wondering if folks have any real life experience or advice to share.

Depends greatly on your use cases. We use more projects. For example, we have a few really-really-global resources (admin VPCs and VPC peering, an AD instance for the dev and infra users, etc.) that go in one truly-global project with just 2 stacks (experimental and global); env-wide global resources (e.g. RDS instances, LBs, etc.) that go in a "shared resources" project with a stack per env (dev1, dev2, citest, usertest, prod, etc.), and an app project that has one stack per app instance (and each app instance knows about the global project/stack, and one stack from the shared resources project). Works for us.

There's a couple of issues asking about this: https://github.com/pulumi/pulumi/issues/7048 https://github.com/pulumi/pulumi/issues/8402

@echoing-dinner-19531 thanks. We’re “lucky” that duplicating the folders/YAML files/etc is not too bad for us, they’re relatively spartan in that sense.

Do you mean, two stacks in the same project have no code in common? That doesn't sound desirable...

Sorry, I mean they don’t have any resources or dependencies in common. So they provision everything from scratch, basically. Their code is 100% shared (coming from our common library).

Ah. Well if there's no difference between the code in infra-dev-project and infra-prod-project, my preference would be to have them in the same project. Anything else risks drift.

I guess in my mind,

git

branches seem like not a good fit, especially for trunk-based development. An example operation is “I want to update all the production infra stacks” - that now involves me somehow deciding from the stack what is a production one and what isn’t, as opposed to just listing all the stacks in the

infra-prod

project. I can of course add some custom config or use naming convention for this, but it does add complexity.

trunk-based development is separate from using branches for deployment / release strategies. But point taken, you will need some scripts to manage simultaneous multi-target deployments. It is possible to use Pulumi + automation-api to create tools to do exactly what you want, but someone has to pay for that dev effort...

Yes. We’re using automation API already (mostly to automate the creation and deletion of stacks without needing to do bash scripting for them), so it’s not unreasonable to add this there.

One of the big benefits of moving your infrastructure declarations into source control with IaC is that it enables you to leverage your branching or your release flow to promote your infrastructure changes from dev to staging to prod the same way you would any other part of your release cycle. Splitting different environments into different Pulumi projects just sounds to me like you're making it harder for yourself. If you are leveraging source control, it is trivial to make an infrastructure change to only your staging stacks when that change resides only on your staging branch and not on your production branch yet. The stack requires no knowledge of what environment it is deploying. Even with trunk-based development, you can still control what commit or changeset or tag your Pulumi deployment is executing on. Maybe I'm still just not grokking the problem you're trying to solve.

I’m trying to think how to best explain without boring everyone with very specific details 🙂

Ok I kinda see now. FYI Automation API can create a Pulumi project on the fly - you just need to point it to the right TStack implementation to deploy. What that means is that your source control could be singular

/infra

while your pulumi state backend could reflect

/infra-{env}

- so you could simulataneously have no code duplication in your repository and split your environments into different pulumi projects from the perspective of the state

Interesting - I didn’t know that. We’ve kept it so that even though we use the Automation API, everything is still runnable with

pulumi up

and so on, to make it easier to do iterative dev work on the stack itself.