What are the best resources on organizing pulumi code? I found the official guide lacking. I want to have a hierarchy of configs which is discussed here but seems left unanswered on what good options would be. I didn’t find out if Pulumi.yaml and Pulumi.stack.yaml will read from stack first then project. Another major question is what is the recommended way to split in multiple files. Examples show single TS file. Challenge being dependencies. I can figure it out but seeing what others are doing would be nice.

On a global level, you can make use of different projects/stacks and use

StackReferences

to import values/ids created in some base infra stack e.g. See Organizing projects and stacks for more details. If it comes to structuring your project files, it’s not just about putting some helper functions into different

.ts

files. Pulumi comes with the concept of resources, and especially `ComponentResource`s that help building reusable components. Those classes create some own scope by parenting the regular resources and providing them a set of `Provider`s to be used. That way you can create a set of resources (inside one

ComponentResource

that use a specific set of providers without specifying the provider for each individual resource. You can have a look at the docs on ComponentResources for more details. The pulumi/examples repo has some samples for using components, for example this webserver component or some more using the search… I personally really like modularizing using `ComponentResource`s on a project level and I see some benefits of `StackReference`s if there are really global resources like DNS or VPCs, but I definitely would not create too many stacks upfront because this will create some new issues of outputs management and execution orders.

Hrm interesting. I’ve actually been breaking up my stacks and projects fairly fine-grained. Currently one per storage service (e.g. Cassandra, Elasticsearch, etc).

@rhythmic-finland-36256 let’s say I need to create a project with several resources and they won’t be used often enough to justify a ComponentResource. How do I split up my code in ways that makes sense, like a separate file for network resource, and separate file for storage resources. Is breaking it down into functions called from index.ts the recommended approach?

@narrow-area-20379 what do you want to support? Multiple environments, stateless vs stateful workloads, etc? I don’t think there’s a 1 size fits all approach. If the workloads are stateless more monolithic is probably fine, stateful I’d go more granular to prevent unintended consequences for a datastore.

@narrow-area-20379 I didn’t have any technical issues with pulumi using multiple stacks. References can be templated, e.g. importing the same stack name from a different project - all that works pretty nicely (as long as you have a 11️⃣1:1 relation or you share one global instance, e.g. your main networking stack). When I started, I created one project for the main infrastructure (like networks, public IPs, …), one project for the Kubernetes cluster, one for basic stuff to install into the cluster (ingress-controller, cert-manager, …) and one for the real application. But at some point I realized that I need an additional public IP for a non-http service I want to expose for one specific deployment. So where to put that now? Putting it into the app stack somehow ends with public IPs defined in different places, putting it into the basic infra project means that the upper projects (apps) leak into the basic infrastructure. That’s why I refactored the projects to be more app-specific which means having all stuff required to run a specific app in one project (also the cluster and the specifics like public IPs required for the apps). VPCs, DNS, container registries can still be in some more central project, but the app specific parts I like to have in one place. Using ComponentResources, reusable modules and splitting things up into files really helps keeping an overview without introducing the runtime complexity of executing one project before another. That’s my current opinion, which might change after having a deeper look into the pulumi automation API…

@rhythmic-finland-36256 separate files would still be functions? In my example I have a provider that needs to be passed for every single resource being created. As you cannot override the provider globally 😕 Handling Output type is also not been fun. Learning curve here is harder than I had with Terraform.

I personally use ComponentResources heavily, also for smaller things to avoid exactly that repetition of providers. Best example are Kubernetes deployments that need the provider pointing to a cluster created just before (and thus not available on a global level like things the providers configured in the pulumi config). For me that encapsulation works pretty well. But there are also people using plain functions or even creating resources by just importing another file. I like the more explicit way of a resource I instantiate. Regarding outputs, those can also be bundled as meaningful objects inside such a resource, so you don't need to export all values one by one. That's just my personal preference and experience and maybe not how everybody solves it. Does that sound reasonable?