Channels
pulumi-ai
package-authoring
pulumi-cdk
oracle-cloud-infrastructure
learn-pulumi-events
linen
registry
built-with-pulumi
pulumi-cloud
contribex
testingtesting321
hacktoberfest
cloudengineering
yaml
pulumi-crosscode
content-share
finops
multi-language-hackathon
office-hours
workshops
blog-posts
localstack
welcome
gitlab
pulumi-kubernetes-operator
jobs
aws
pulumi-deployments
dotnet
golang
announcements
java
pulumiverse
python
install
getting-started
cloudengineering-support
testingtesting123
hackathon-03-19-2020
typescript
google-cloud
contribute
azure
kubernetes
docs
automation-api
status
general
Title
a
acceptable-plumber-31485
03/14/2023, 3:01 PMWe have several AWS accounts, one per each environment(dev, stage, prod, and few more). We know that we can have many resources per account. In Terraform, we have a single git project per AWS account. The git project file structure is structured by region like us-east-1, us-west-1, etc. Inside each directories, we have files per type of resource. For example, anything related to a dns, changes must be done in route53dns.tf. If it's related to s3 bucket, all changes must go to s3.tf. This whole git project uses one state file and is located in an s3 bucket. That means, there are other s3 buckets for other terraform projects in other environments.
I've been reading Pulumi for the past few days and it looks like the state in Pulumi is called stacks. Am I right that I should create 1 stack for all resources per one AWS account? I'm also interested in how you designed your file structure for different resources.
l
little-cartoon-10569
03/14/2023, 7:52 PMYou should (usually) group your resources into projects defined by their deployment cycle. The account and region is less important, because you can have multiple providers in the one project, allowing you to deploy to multiple places in the same
pulumi upEach project should have stacks based on repeated instances of your project. This may be equivalent to account+region for you, or just account, but most likely, you should think about dev, stage, prod, etc.
a
acceptable-plumber-31485
03/14/2023, 7:53 PMWhen you say
deploy to multiple places in the samel
little-cartoon-10569
03/14/2023, 7:54 PMYes
a
acceptable-plumber-31485
03/14/2023, 7:54 PMI was thinking of doing separate deployments so that if something is wrong with the application/resource on STAGE, we will hold off in deploying it to PROD account
l
little-cartoon-10569
03/14/2023, 7:54 PMThere's more explanation (and alternative approaches) here: https://www.pulumi.com/docs/guides/organizing-projects-stacks/
a
acceptable-plumber-31485
03/14/2023, 7:54 PMthank you
l
little-cartoon-10569
03/14/2023, 7:55 PMI was thinking of doing separate deployments so that if something is wrong with the application/resource on STAGE, we will hold off in deploying it to PROD accountHence, different stacks.
a
acceptable-plumber-31485
03/14/2023, 7:55 PMgot it
l
little-cartoon-10569
03/14/2023, 7:55 PMAnd in STAGE, you might want to deploy the database once only, and the app once peer week. So, you'd need 2 different projects.
a
acceptable-plumber-31485
03/14/2023, 7:55 PMgot it
we have 2 web teams that build different applications. In our current tooling, they are saved as a separate artifact, totally no relationship with each other. In pulumi, that means there will be 2 or more stacks per team and those stacks won't be related to the other teams. Correct?
l
little-cartoon-10569
03/14/2023, 8:00 PMNo, probably not. The two apps probably don't get deployed at the same time or at the same frequency, so different projects would suit. Each project would have its own DEV, STAGE, PROD etc.
a
acceptable-plumber-31485
03/14/2023, 8:01 PMgot it!
l
little-cartoon-10569
03/14/2023, 8:02 PMAssuming that both teams deploy on top of the same shared infrastructure (into the same VPC, using the same Redis and Aurora, etc.) then you'll need a different project for the shared infra, too.
Maybe more than one.
a
acceptable-plumber-31485
03/14/2023, 8:02 PMI see. I haven't seen that kind of scenario yet at work
but good to know
l
little-cartoon-10569
03/14/2023, 8:03 PMDNS and VPC are rarely maintained the same project as an app.
Well, some DNS is, some isn't 🙂 The problem, as we all know, is usually DNS.
a
acceptable-plumber-31485
03/14/2023, 8:04 PMfor us, dns is always updated especially when a new project is created by our users. Route53 will have a new entry on the private zone specifically an A and CNAME record of the new app deployed
it's automatically updated by our tooling called Foremast, which I'm planning to replace with Pulumi 🙂
l
little-cartoon-10569
03/14/2023, 8:05 PMBut for your TLS certs for you main domain, you won't want to update those every time you update an application. And TLS certs need DNS records...
a
acceptable-plumber-31485
03/14/2023, 8:05 PMyes
l
little-cartoon-10569
03/14/2023, 8:05 PMSo that would be in a shared project, not an app project.
a
acceptable-plumber-31485
03/14/2023, 8:05 PMyep
I just started using Pulumi 4 nights ago. I'm still looking for equivalent process we have now but using pulumi. I was told some doesn't exist yet. Looks like I'll be building our own code for hiding pulumi to our developer teams. That way, we can keep existing configuration files(read by Foremast) in devteams' git repos and have my new code read those configuration files and pass it to Pulumi to build resources
l
little-cartoon-10569
03/14/2023, 8:09 PMYou can do that. Automation-api can help with that. Or just hide it within your build system.
yarn run deploypulumi upOr
yarn run deploy STAGEa
acceptable-plumber-31485
03/14/2023, 8:09 PMyep, that's one way
I'm thinking of creating a docker image where I will use it with gitlab-runner. This image will have the pulumi tools installed
l
little-cartoon-10569
03/14/2023, 8:10 PMAlso, don't feel you have to hide anything. Coders love new tricks and tools 🙂
Those docker images already exist
Pulumi releases many docker images, per language.
a
acceptable-plumber-31485
03/14/2023, 8:10 PMyeah that's true but i'm worried they will have so much control
l
little-cartoon-10569
03/14/2023, 8:10 PMYou can review the Dockerfiles. It's safe
Oh, you mean the coders will have control?
a
acceptable-plumber-31485
03/14/2023, 8:11 PMyep, I'll definitely use it 🙂
yeah coders wil have control
that's why we hid Foremast from them
l
little-cartoon-10569
03/14/2023, 8:11 PMCoders should have the control. Otherwise your lead times will grow and frustration will rise.
You just need to build the right safety into the system so that control is safe.
a
acceptable-plumber-31485
03/14/2023, 8:12 PMso far it's working and it's been like this for 5+ years now. They do have some annoyance 😄
especially our team is small now, their feature requests doesn't get created quickly
I'll read about automation-api today
l
little-cartoon-10569
03/14/2023, 8:13 PMThen they're probably too big.. but that's a story for another day.
a
acceptable-plumber-31485
03/14/2023, 8:13 PMyep
I have a question about resources and state. So I've been playing with pulumi locally on a self managed gitlab(in a docker container). Am I right that the state is being saved locally?
l
little-cartoon-10569
03/14/2023, 8:16 PMThat's configured by your backend (in Pulumi.yaml), which is configured by your
pulumi logina
acceptable-plumber-31485
03/14/2023, 8:17 PMlet me check
name: iac-workshop-ec2-webservers
runtime:
name: python
options:
virtualenv: venv
description: A minimal AWS Python Pulumi programthat's my yaml file
l
little-cartoon-10569
03/14/2023, 8:19 PMYou're using the Pulumi backend.
It's the default.
a
acceptable-plumber-31485
03/14/2023, 8:20 PMI have not read about backends yet. Is backend related to storage?
and is Pulumi backend locally saved ?
l
little-cartoon-10569
03/14/2023, 8:21 PMBackend is storage of state. Pulumi state is just a JSON file. The Pulumi service stores state for you on Pulumi infrastructure. You need to log into a local backed if you want to store it locally.
a
acceptable-plumber-31485
03/14/2023, 8:21 PMgot it
let me check if I'll be able to find the json file on the directory
can't find it on the directory, lol
I only found "./venv/lib/python3.10/site-packages/pulumi_aws/pulumi-plugin.json"
looks like I found it
it's in
vi ~/.pulumi/stacks/dev.jsonOnce pulumi gets implemented in production at work, I'll have to switch the state from local to s3. It's because the machine we use for running CI is terminated after the ci scripts are done
Now I am thinking, looks like backend is the first thing to implement