https://pulumi.com logo
Join Slack
Powered by
anyone know the right way to share resources betwe...
# general
g
anyone know the right way to share resources between stacks in pulumi? for e.g. i have two stacks 
staging
and 
prod
i want to create an IAM user in aws that is shared/known to both environments. so when i deploy the 
staging
stack for example, it'll create this user, but then when i switch to 
prod
, i want it to recognise the user has already been created, rather than trying to re-create it which is what happens given the resource isn't being shared between the stacks any help would be much appreciated as I'm new to pulumi
l
d
For shared resources, it'd be best to setup another project, then have both your staging/prod stacks in your current project reference them using Stack References
g
thanks, came across this. just need to figure out/see examples for how to effectively structure the project folders/files
d
There's a quick guide to help with deciding overall structure here: https://www.pulumi.com/docs/using-pulumi/organizing-projects-stacks/
g
@dry-keyboard-94795 okay. ideally i want to keep it all contained within a single project. just skimmed through that article. im thinking infra • aws ◦ shared ▪︎ index.ts ◦ staging ▪︎ index.ts ◦ prod ▪︎ index.ts guessing this will work, if i utilise stack refs to bring into 
prod
and 
staging
from 
shared
d
I'd expect the staging + prod stacks to be the same project
g
they would all be in the same Pulumi project if that's what you mean? e.g. they all use the same AWS account. i think it makes sense to keep them separate? that way I can for example define a lambda that will only be deployed into staging and not in prod?
d
For simplicity, yes you could deploy all services from a single project. Instead of having separate staging/prod entrypoints, I'd recommend using feature flags; ie your 
Pulumi.staging.yaml
file contains 
enable-some-service: true
. This way, your code is the same between production + staging
s
+100 to everything @dry-keyboard-94795 said. 😄
g
okay just to make sure ive got this clear youre suggesting one stack for staging & prod but making use of feature flags set in the config to manage differences between them?
d
Multiple stacks, one project. A project is the directory that holds your code (
Pulumi.yaml
index.js
). A Stack is represented by the config 
Pulumi.staging.yaml
(for the staging stack)
s
Put another way: one project for staging and prod, but each of them would be a separate stack within that project (and here I'm using "project" and "stack" in the Pulumi sense).
The IAM user would be a separate project with its own stack, and the staging and prod stacks would reference that user via a stack reference.
d
Just on the shared user, I'm curious on your usecase. It's not necessarily wrong to do, I've done similar to shortcut setup for CI/CD before
g
hmm, i understand, ideally id like one project with 3 stacks 
staging
, 
prod
and 
shared
and 
staging
and 
prod
can make use of stack references from 
shared
, rather than having a separate project for 
shared
d
It will be simpler to just have a separate 
shared
project, so your code doesn't need to be concerned with too much branching. You could even nest it in the same directory if you want: • Infra/ ◦ Pulumi.yaml ◦ shared/ ▪︎ Pulumi.yaml
g
@dry-keyboard-94795 essentially there are some resources we have on AWS that aren't necessary for us to have per environment. e..g we're happy to use the same IAM user for both 
prod
and 
staging
resources. we have a few instances of those. because of that we just want to shared resources so there isnt conflict (e..g prod does not try and recreate that IAM user). hopefully that makes sense?
@dry-keyboard-94795 thats a good idea and structure
s
hmm, i understand, ideally id like one project with 3 stacks
@gorgeous-pharmacist-71907 You could make it work that way, but IMO it would be unnecessarily complex. You want to think of a stack as an instance of the infrastructure defined in your program. This is what enables you to have identical (or nearly identical) infrastructure in your staging environment (one stack) and your production environment (a second stack)---because they run the same code, but with independent state and independent configuration values. As Anthony mentioned, it's best if you have a separate 
shared
project (and every project will have at least one stack) when you have resources that need to be accessed/used/referenced by other stacks (like the IAM user in your use case).
But listen to Anthony---he's smart and will definitely lead you in the right direction. 👍🏻
g
yeah so if ive understood correctly, i think we're on the same page now @salmon-account-74572? so having two projects. one project housing shared infra. the other housing code for both 
prod
and 
staging
(stack for each for independent config)
s
Yep! And to Anthony's point earlier, you can write the code for the 
prod
and 
staging
stacks such that they can use configuration values to provision slightly differently (for example, maybe you want to use smaller instance types in staging, so parameterize that as a configuration value).
d
^^ yep. This will help in the longer term, as iam management, + maybe core networking, requires more privileges than an app deployment. Keeping them split will reduce the temptation to just use admin roles for deployment
g
yeah i really like this setup, sounds like it ticks all the boxes and quite an easy workflow to understand/maintain, single repo is always nice too
thanks a lot @dry-keyboard-94795 @salmon-account-74572 👏
s
Happy to help!
g
sorry to message again, i've got a test project up and running but it seems as though it's not recognising the stack reference created a 
shared
project & stack as we mentioned & exported an output named 
IAMUserARN
and verified with 
pulumi stack output
then in my 
index.ts
for the 
staging|prod
stack I've setup, i have the following:
Copy code
const sharedStack = new Pulumi.StackReference("shared")

sharedStack.getOutput("IAMUserARN").apply((arn) => {
    console.log(`${arn}`)
})
when i run 
pulumi up
, it doesnt print anything: am i doing something wrong here?
i've tried 
<http://Pulumi.log.info|Pulumi.log.info>
too and i get the same result
d
The stack reference doesn't look right, it should include a project name and the stack name
And an org name if you're using orgs with pulumi cloud
g
yeah so im using 
file://
to store pulumi state i came across this SO post which recommended setting it to just the name of the stack: https://stackoverflow.com/questions/71491871/how-to-reference-another-stack-for-file-backend
d
Ah, I'm not familiar with our stack references work for the file backend
g
ah fair enough, yeah it's a tricky one, do you recommend i ask in the general chat or raise an issue on Github? ive scoured google a fair bit and can't see anything on it..
d
So you have 
backend.url
set in 
Pulumi.yaml
?
It looks like you still need to specify both the project and stack name in the reference, but should work
g
i didn't know i had to set 
backend.url
, thought 
pulumi login file://./
was enough in any case i've added that now just to clarify my project structure is: infra • shared ◦ index.ts ◦ Pulumi.yaml ◦ Pulumi.shared.yaml • resources ◦ iam.ts ◦ ... • index.ts • Pulumi.yaml • Pulumi.staging.yaml • Pulumi.prod.yaml i set the 
backend.url
but that didnt seem to work i also tried 
shared/shared
as you suggested. the contents of my 
Pulumi.yaml
file in the shared directory is:
Copy code
name: shared
runtime: nodejs
description: Shared infrastructure that can be used in other stacks
organization: testorg
backend:
  url: file://./
im guessing that's correct, not sure if ive missed anything
d
the backend.url will need to point to the same location in both projects
g
oh okay, let me try that
d
If it was unset for both, this would have been the case already, and using 
shared/shared
as your stack reference will work
g
okay i think it worked in the 
infra
root is where im storing the state so in 
infra/shared/Pulumi.yaml
, i set the backend url to 
file://../
and in 
infra/Pulumi.yaml
, i set the backend url 
file://./
so it points to the same dir and then weirdly i tried 
shared/shared
but it demanded 
organization
at the beginning. changing to 
organization/shared/shared
seems to have worked
s
I'll admit that stack references outside of Pulumi Cloud aren't documented well; it's definitely an area where we can provide more examples. That being said, I'm glad it seems to be working for you now.
d
Support was only added in the end of March. I don't see many using object/file based state either
Just so you know, by default, secrets won't be encrypted with file-based as far as I know, so be wary of committing the state directory. Also be wary of losing that directory. Going with pulumi cloud is the simplest way of setting up for production, though you can use s3
s
Going with pulumi cloud is the simplest way of setting up for production
100 rainbow
g
yeah that makes sense, im just testing around atm before committing to anything
but good to know
one other q, do you know if its possible to export a provider and use it? eg resources accept the 
{provider}
param but the type is 
Pulumi.ProviderResource
. however the 
sharedStack.getOutput("euWest1Provider")
wont return that type
d
It won't, you need to output the parameters from the stack, then initialise again in the receiving stack
g
okay that makes sense, ill try that
thanks
turns out 
Pulumi.ProviderResource
is an abstract class so no luck there and also cant use 
new AWS.Provider
as that'll just create another provider in the receiving stack
d
Creating a provider is the intended usecase
It doesn't create anything in your aws account, it's just configuring auth + how to connect
g
okay in that case no need to create in the shared then
d
Pulumi provides you a defualt provider, that can be configuring using the config file, or environment variables
It's best to use environment variables, so you're not storing any secrets in files
g
yup so setup the default one but theres cases where we want to create infra in different regions and so want to manually set that per resource
d
Correct
i
interesting discussion on the need to create 2 projects, imo I'd create one project -- I don't see any downside to using different programs for different stacks in the same project file structure is easy enough, you just have a dedicated folder for each program type. multiple folders can share the same 
Pulumi.yaml
project name and if it's important to distinguish which program was used to create which stack, you can export static identifiers from your program
I do this currently, where I have a lot of stacks in one project, using a different program for each microservice. The naming convention is 
[service].[env]
keen to understand if there's anything I'm missing that might come to bite me later on down the road...
3 Views
Open in Slack
PreviousNext
Powered by