Hi, I'm trying to deploy several k8s services using pulumi and I'm having some difficulties wrapping my head around the parent and dependency concepts. My setup is a k3s cluster with 1 master node and 3 worker nodes. For simplicity my setup is as follows, an infrastructure part and an application part. The infrastructure consists of two services, Calico and metallb. The application is ESPhome. In my real test setup there are more infrastructure services, but for my problem they are not relevant. The natural flow of the deployment is to have Calico deployed, then metallb and finally ESPhome. I would like to have this deployment done in stages, so first Calico is deployed and when all resources (pods, services, etc) are ready then next stage is deployed so forth. I would also like to have related resources grouped in a class, e.g. the calico installation is two yaml files from github and two configuration objects. I have tried to group the four pulumi resources for Calico into a class and setup depends_on and parent in various ways but the result is always the same. Some Calico resources are being deployed, and some are in pending/starting and then the next stage is started. The result is that some of the later stages are failing because infrastructure resources (crds, services) haven't been deployed yet. I haven't been able to find a "real life" example of deploying a larger setup like this. Only the "toy"/getting started examples and they don't seem to highlight the problems that I'm facing. So my questions are, • How would you structure this? Am I complete gone off-track? • Are there more "real life" examples that I can get inspired from? BTW. I'm using the python SDK. And finally I'm terribly sorry for the many edits this post already have gotten from my side... "Shift+Enter" is natural for me 😄

For larger infrastructure, it typically makes sense to have several projects. For example, in your case: • infrastructure -> cluster nodes and basic services (calico and metallb) • application -> ESPhome You could also split "infrastructure" into a "(virtual) metal" part and a "foundation" project. The "application" project uses a stack reference to fetch the kubeconfig that the "infrastructure" project exports (e.g., the "application" dev stack fetches the "infrastructure" dev stack's kubeconfig) and otherwise assumes that this will provide it access to a Kubernetes cluster with calico and metallb up and running. In the spirit of "things that change together go together," splitting your IaC into projects allows you to evolve and deploy them separately. Usually, the "infrastructure" parts don't change often, so there's no need to run through a state refresh for them every time you want to deploy an update to your application. (When working in a team, this is also the only reliable way to give members control over only "their" parts of the deployment.) Creating and maintaining a Kubernetes cluster and deploying an application are two separate activities, the application should not have to care about the cluster's internals as much as possible. In general, I find that when I'm starting to add lots of manual dependency declarations and trying to find workarounds for resources to be deployed in the correct sequence, this is a sign that I should split my infra code into separate projects. In small setups, you can still always deploy both projects in sequence, but you ensure that your "infrastructure" is ready by waiting for its "up" to complete, rather than trying to achieve this through dependency declarations within the code. https://www.pulumi.com/docs/iac/using-pulumi/organizing-projects-stacks/ has more background info and examples.

I've seen setups with Pulumi where there almost all code was in one huge stack with 700+ resources, for very different lifecycles and purposes (some infra, some app related). This raises the issue of how to communicate values between projects/stacks. I like to have a separate parameterisation system with templating across all projects, so that predictable values like resource names, domain names, etc, can be passed to all projects. For non-predictable values such as resource IDs, and system-allocated domains (e.g. AWS RDS hostnames, or account IDs), I like to tag them with a predictable name, then write a helper function to find a resource based on that name. In some cases where this is impractical, saving the generated values in AWS Parameter Store or similar would work too. Overall, decoupling projects, as with software modules generally, will help future evolution, and is worth going for as long as the required code remains relatively simple. I've just realised I'm reviving an old thread here, sorry! I'm curious to see how @refined-furniture-67680 solved this.