Hey :wave: I am interested in implementing Pulumi ...
# general
m
Hey šŸ‘‹ I am interested in implementing Pulumi projects and stacks and wanted to introduce a super "DRY (Do not Repeat Yourself)" setup that takes advantages of different stacks. Context on our use case for DRY: We have customers that we provision resources for. These resources are the same for all customers (with some variable differences), each customer will have 1 or 2 environments (I.E sandbox and/or production). Each environment is the same with some variable differences. I would ideally like each component for that environment (i.e VPC) to be a stack. I would like to have a single repository that holds the code for all the customers, but each customer to be its own project. To try and visualise an ideal repo setup:
Copy code
.
ā””ā”€ā”€ customer
    ā”œā”€ā”€ common
    ā”‚   ā””ā”€ā”€ route53
    ā”‚       ā””ā”€ā”€ main.go
    ā”œā”€ā”€ env
    ā”‚   ā”œā”€ā”€ eks
    ā”‚   ā”‚   ā””ā”€ā”€ main.go
    ā”‚   ā””ā”€ā”€ vpc
    ā”‚       ā””ā”€ā”€ main.go
    ā”œā”€ā”€ pulumi.customerA.yaml <--- Variables that define some customer values and both sandbox and production values.
    ā””ā”€ā”€ pulumi.customerB.yaml
How would I go about this, since I don;t see how I can make this work with Pulumi?
m
The project file,
Pulumi.yaml
has a
config
section that can be applied to all stacks. You can override these values in the stack
pulumi.customerA.yaml
files as well.
If you're saying these customers have different variables, but otherwise are deploying the same resources, I don't think each customer should be its own project, but rather their own stack. https://www.pulumi.com/docs/concepts/#how-does-pulumi-work Where "stacks" are instances of a project.
m
Yeah I think having a customer on its own stack is a better idea. However, how would a single stack for a customer perform when there are ~50 packages being used per environment, and if the customer has 2 environments it will be referencing 100 packages. Would this slow down the speed of making a changes. For example, I am updating a s3 bucket to add a policy, would the full stack need to perform a refresh just for a policy update to a bucket?
That is why I initially thought having a stack per component I.e VPC is a better idea for speed
m
I'm not sure I follow. By packages, what are you referring to? Each stack would have its own resource graph, e.g. if each customer has their own s3 bucket, each stack has an s3 bucket, and a policy update would have to happen to all of them. These deployments could be run in parallel. If you are saying you have shared resources between stacks, that's a slightly different pattern you're asking for, and you want a "shared" infrastructure stack that other "child" stacks can reference.
m
> By packages, what are you referring to? I am still new to the Pulumi lingo, but I am using package in replacement of module as you would in terraform (if you are familiar with tf)
m
You may want to check out these documents if you haven't already, they may answer some of your questions, and provide examples for your use case. ā€¢ https://www.pulumi.com/docs/concepts/ ā€¢ https://www.pulumi.com/docs/using-pulumi/organizing-projects-stacks/ ā€¢ https://www.pulumi.com/blog/iac-recommended-practices-code-organization-and-stacks/
One of the beautiful features of Pulumi is the native integration with the programming language's ecosystem. So if you use Python for example, you could distribute your own pulumi packages as a python package to keep things dry.
You wouldn't have to copy and paste it 50 times, 1 for every customer.
m
Thank you, I will definitely review the links you have provided.
m
That being said, I think if you read through the various use cases and options, you might find a pattern that already fits your needs within the Pulumi constructs, e.g. projects/stacks.
m
Might be a stupid question, but would a stack need to be in a single folder?
Or, could I still use the folder structure I shared above to define a single stack?
m
The minimum/expected configuration of a stack is just the configuration, e.g. your customer YAMLs. So I think your layout works within Pulumi design concepts, yes.
oh, but I would not have the top level "customer" directory, unless that's a project within a monorepo
m
Cool, in that case if I were to make a change to
eks
will pulumi only preview/apply eks?
m
Pulumi, like terraform uses a "desired state" model, so it compares the previous state to the new desired state, calculates the difference and uses the providers to apply the changes (yes, only the what has changed). Because of the asynchronous design, and inferred, or explicitly declared relationships in the resource graph, these operations are optimized to run in parallel by the Pulumi engine.
m
Thank you for the help! I might be back with more questions šŸ˜