https://pulumi.com logo
Title
a

acceptable-plumber-31485

03/14/2023, 3:01 PM
We have several AWS accounts, one per each environment(dev, stage, prod, and few more). We know that we can have many resources per account. In Terraform, we have a single git project per AWS account. The git project file structure is structured by region like us-east-1, us-west-1, etc. Inside each directories, we have files per type of resource. For example, anything related to a dns, changes must be done in route53dns.tf. If it's related to s3 bucket, all changes must go to s3.tf. This whole git project uses one state file and is located in an s3 bucket. That means, there are other s3 buckets for other terraform projects in other environments. I've been reading Pulumi for the past few days and it looks like the state in Pulumi is called stacks. Am I right that I should create 1 stack for all resources per one AWS account? I'm also interested in how you designed your file structure for different resources.
l

little-cartoon-10569

03/14/2023, 7:52 PM
You should (usually) group your resources into projects defined by their deployment cycle. The account and region is less important, because you can have multiple providers in the one project, allowing you to deploy to multiple places in the same
pulumi up
.
Each project should have stacks based on repeated instances of your project. This may be equivalent to account+region for you, or just account, but most likely, you should think about dev, stage, prod, etc.
a

acceptable-plumber-31485

03/14/2023, 7:53 PM
When you say
deploy to multiple places in the same
did you mean, multiple AWS accounts?
l

little-cartoon-10569

03/14/2023, 7:54 PM
Yes
a

acceptable-plumber-31485

03/14/2023, 7:54 PM
I was thinking of doing separate deployments so that if something is wrong with the application/resource on STAGE, we will hold off in deploying it to PROD account
l

little-cartoon-10569

03/14/2023, 7:54 PM
There's more explanation (and alternative approaches) here: https://www.pulumi.com/docs/guides/organizing-projects-stacks/
a

acceptable-plumber-31485

03/14/2023, 7:54 PM
thank you
l

little-cartoon-10569

03/14/2023, 7:55 PM
I was thinking of doing separate deployments so that if something is wrong with the application/resource on STAGE, we will hold off in deploying it to PROD account
Hence, different stacks.
a

acceptable-plumber-31485

03/14/2023, 7:55 PM
got it
l

little-cartoon-10569

03/14/2023, 7:55 PM
And in STAGE, you might want to deploy the database once only, and the app once peer week. So, you'd need 2 different projects.
a

acceptable-plumber-31485

03/14/2023, 7:55 PM
got it
we have 2 web teams that build different applications. In our current tooling, they are saved as a separate artifact, totally no relationship with each other. In pulumi, that means there will be 2 or more stacks per team and those stacks won't be related to the other teams. Correct?
l

little-cartoon-10569

03/14/2023, 8:00 PM
No, probably not. The two apps probably don't get deployed at the same time or at the same frequency, so different projects would suit. Each project would have its own DEV, STAGE, PROD etc.
a

acceptable-plumber-31485

03/14/2023, 8:01 PM
got it!
l

little-cartoon-10569

03/14/2023, 8:02 PM
Assuming that both teams deploy on top of the same shared infrastructure (into the same VPC, using the same Redis and Aurora, etc.) then you'll need a different project for the shared infra, too.
Maybe more than one.
a

acceptable-plumber-31485

03/14/2023, 8:02 PM
I see. I haven't seen that kind of scenario yet at work
but good to know
l

little-cartoon-10569

03/14/2023, 8:03 PM
DNS and VPC are rarely maintained the same project as an app.
Well, some DNS is, some isn't 🙂 The problem, as we all know, is usually DNS.
a

acceptable-plumber-31485

03/14/2023, 8:04 PM
for us, dns is always updated especially when a new project is created by our users. Route53 will have a new entry on the private zone specifically an A and CNAME record of the new app deployed
it's automatically updated by our tooling called Foremast, which I'm planning to replace with Pulumi 🙂
l

little-cartoon-10569

03/14/2023, 8:05 PM
But for your TLS certs for you main domain, you won't want to update those every time you update an application. And TLS certs need DNS records...
a

acceptable-plumber-31485

03/14/2023, 8:05 PM
yes
l

little-cartoon-10569

03/14/2023, 8:05 PM
So that would be in a shared project, not an app project.
a

acceptable-plumber-31485

03/14/2023, 8:05 PM
yep
I just started using Pulumi 4 nights ago. I'm still looking for equivalent process we have now but using pulumi. I was told some doesn't exist yet. Looks like I'll be building our own code for hiding pulumi to our developer teams. That way, we can keep existing configuration files(read by Foremast) in devteams' git repos and have my new code read those configuration files and pass it to Pulumi to build resources
l

little-cartoon-10569

03/14/2023, 8:09 PM
You can do that. Automation-api can help with that. Or just hide it within your build system.
yarn run deploy
hides
pulumi up
just fine 🙂
Or
yarn run deploy STAGE
...
a

acceptable-plumber-31485

03/14/2023, 8:09 PM
yep, that's one way
I'm thinking of creating a docker image where I will use it with gitlab-runner. This image will have the pulumi tools installed
l

little-cartoon-10569

03/14/2023, 8:10 PM
Also, don't feel you have to hide anything. Coders love new tricks and tools 🙂
Those docker images already exist
Pulumi releases many docker images, per language.
a

acceptable-plumber-31485

03/14/2023, 8:10 PM
yeah that's true but i'm worried they will have so much control
l

little-cartoon-10569

03/14/2023, 8:10 PM
You can review the Dockerfiles. It's safe
Oh, you mean the coders will have control?
a

acceptable-plumber-31485

03/14/2023, 8:11 PM
yep, I'll definitely use it 🙂
yeah coders wil have control
that's why we hid Foremast from them
l

little-cartoon-10569

03/14/2023, 8:11 PM
Coders should have the control. Otherwise your lead times will grow and frustration will rise.
You just need to build the right safety into the system so that control is safe.
a

acceptable-plumber-31485

03/14/2023, 8:12 PM
so far it's working and it's been like this for 5+ years now. They do have some annoyance 😄
especially our team is small now, their feature requests doesn't get created quickly
I'll read about automation-api today
l

little-cartoon-10569

03/14/2023, 8:13 PM
Then they're probably too big.. but that's a story for another day.
a

acceptable-plumber-31485

03/14/2023, 8:13 PM
yep
I have a question about resources and state. So I've been playing with pulumi locally on a self managed gitlab(in a docker container). Am I right that the state is being saved locally?
l

little-cartoon-10569

03/14/2023, 8:16 PM
That's configured by your backend (in Pulumi.yaml), which is configured by your
pulumi login
. What backend have you logged into?
a

acceptable-plumber-31485

03/14/2023, 8:17 PM
let me check
name: iac-workshop-ec2-webservers
runtime:
  name: python
  options:
    virtualenv: venv
description: A minimal AWS Python Pulumi program
that's my yaml file
l

little-cartoon-10569

03/14/2023, 8:19 PM
You're using the Pulumi backend.
It's the default.
a

acceptable-plumber-31485

03/14/2023, 8:20 PM
I have not read about backends yet. Is backend related to storage?
and is Pulumi backend locally saved ?
l

little-cartoon-10569

03/14/2023, 8:21 PM
Backend is storage of state. Pulumi state is just a JSON file. The Pulumi service stores state for you on Pulumi infrastructure. You need to log into a local backed if you want to store it locally.
a

acceptable-plumber-31485

03/14/2023, 8:21 PM
got it
let me check if I'll be able to find the json file on the directory
can't find it on the directory, lol
I only found "./venv/lib/python3.10/site-packages/pulumi_aws/pulumi-plugin.json"
looks like I found it
it's in
vi ~/.pulumi/stacks/dev.json
my stack name is dev
Once pulumi gets implemented in production at work, I'll have to switch the state from local to s3. It's because the machine we use for running CI is terminated after the ci scripts are done
Now I am thinking, looks like backend is the first thing to implement