Could you give me some guidance on how to proceed ...
# general
Could you give me some guidance on how to proceed with a mono repo hosting multiple app resources? Right now, the plan is to have a datadog-specific repo that contains all of our datadog resources (integrations, synthetics tests, monitors, etc) with nothing else in the repo. We have about 300 Datadog monitors specific to apps that we need to convert from another tool to Pulumi as part of a PoC against Terragrunt/Terraform. I am testing using Python. From what I have gathered and read, there are a few approaches to deploying this. 1. Lumping all the resources into a single module and using that. This is not desired, as I would like to have separate folders and files within the repo for organization sake to have parity with previous IaC tool strategies. 2. Create a new folder for each app/service. The Pulumi documentation shows that we should configure each folder as its own unique project. The downside here is that CI jobs need to be configured for each project (we are using s3 backend for this PoC), 3. Create a new folder and write logic to crawl each folder and file and execute them in main. This will have nested folders, but be all contained within a single project. 4. Create a new module in the root of the existing project for each app and service, and import each module into for execution, having a "flat" project consisting of 300 files, one for each service. This requires lots of imports and extra code for each new project after it is defined. What is the recommended approach here?
Followup question, using s3 backend, when does size of state start to become an issue? if at all?
From a Pulumi perspective, I would say that #2 is probably the best approach. It’s true that “Pulumi project ~= folder”. This doesn’t preclude you from factoring out common code and re-using that common code across all the Pulumi projects (i.e., putting common code into a
directory or the like and then importing into each Pulumi program as applicable/needed). As for state, I don’t know the answer to any size that might generate a concern, but each stack (an instance of a Pulumi project) will have its own state. Since you are separating things into separate projects/folders, then you’ll have separate stacks and therefore separate state files.
You might also find some of the recommendations in this series helpful:
NP! Let us know if we can help further.