https://pulumi.com logo
Powered by
Title
a

able-camera-57198

11/06/2021, 6:26 PM
Is there any advice around using a single configuration structure and file across multiple stacks? Does pulumi offer any way to pick this up by say... having a stack config file in the parent folder of all my stacks?
g

green-stone-37839

11/06/2021, 8:33 PM
It's possible by passing the location of the configuration file to Pulumi during 
up/preview
. The 
--config-file {location}
flag is what you're looking for. Just curious, why would you want to use the same configuration file for multiple stacks?
a

able-camera-57198

11/06/2021, 8:54 PM
I've got a nice data structure that represents some common intentions/conventions in a multi-region setup.
I have it as a DTO in a common library that all my stacks refer to.
Which is really helpful, 'cause I can then do: 
configuration.GetObject<Project>("project");
And that value is strongly typed with information about all my regions.
What I was hoping to do was avoid repeating that structure once per 
Pulumi.stack.yaml
file. I guess what would be really cool is if I could import another 
.yaml
file. Or alternatively, provide multiple that then get merged together prior to being passed to Pulumi.
myorg-core:project:
    Debug: true
    Name: "myorg-prototype-3"
    OrganizationName: "myorg"
    OrganizationNameShort: "mp"
    EnvironmentNameShort: "mpp3"
    DefaultLocation: "us-central1"
    Regions:
      ca:
        Code: "ca"
        Global: true
        Location: "northamerica-northeast1"
        Cidr: "10.8.0.0/28"
      us:
        Code: "us"
        Location: "us-central1"
        Global: false
        Cidr: "10.9.0.0/28"
  gcp:project: myorg-prototype-3
  google-native:project: myorg-prototype-3
☝️🏻 This is repeated in all of my stacks.
g

green-stone-37839

11/06/2021, 8:59 PM
Pulumi.stack.yaml
files are meant to be per stack, so the configuration isn't really designed to do what you'd like, although it will support it. You're using .net, right? What's stopping you from just using a common appsettings.json file and using .net apis, such as 
ConfigurationBuilder
to use your shared config file?
a

able-camera-57198

11/06/2021, 9:00 PM
Probably nothing, but my default is to try and work within the system before I start doing my own thing 😅
g

green-stone-37839

11/06/2021, 9:00 PM
I hear you, but it's just code, so .net configuration is working within the system
a

able-camera-57198

11/06/2021, 9:01 PM
Alright, so I guess outside of parsing it, pulumi really has no preferences/involvement with the 
yaml
files?
Like it's not expecting them to be there?
Or will I end up with a vestigial 
yaml
AND the config.
g

green-stone-37839

11/06/2021, 9:01 PM
That's true. One sec let me grab another option for you.
👍🏻 1
Disregard, what I was thinking isn't going to support the usecase you want. I think the easiest option for you is to pass the location of the stack configuration file when exeucting an 
up/prevew
. 
pulumi up --config-file ../my-shared-stack-config.yaml
a

able-camera-57198

11/06/2021, 9:08 PM
That's not too too bad.
g

green-stone-37839

11/06/2021, 9:08 PM
that was my original suggestion 🙂
a

able-camera-57198

11/06/2021, 9:08 PM
Hmm....
What about the namespace though?
config:
  my-org-core:project:
g

green-stone-37839

11/06/2021, 9:09 PM
project namespace, is project level not stack
a

able-camera-57198

11/06/2021, 9:09 PM
my-org-core:project:
seems to influence where 
configuration.GetObject<Project>("project");
begins looking.
g

green-stone-37839

11/06/2021, 9:10 PM
Will all your stacks.reside in the same project? Or different projects?
a

able-camera-57198

11/06/2021, 9:10 PM
Different projects.
So each stack is its own 
.csproj
with their own 
Pulumi.yaml
, in sibling folders.
And I guess - at least for the moment - I have a 1:1 correlation between projects and stacks as a result?
g

green-stone-37839

11/06/2021, 9:13 PM
Stacks are meant to be members inside a single project. Think of a project as an application and stacks are the different versions of that application. For instance, dev, test, stage, prod...
a

able-camera-57198

11/06/2021, 9:13 PM
But the stack code can be reused between multiple stacks, right?
(I think of 
public class MyStack : Stack
as more like a "stack template", is that right?)
g

green-stone-37839

11/06/2021, 9:15 PM
But the stack code can be reused between multiple stacks, right?
There is no such thing as stack code. Projects contain code (program.cs, .csproj, etc). Stacks are instances of a project.
a

able-camera-57198

11/06/2021, 9:15 PM
It's not like I have to make: • 
public class MyDevStack : Stack
• 
public class MyProdStack : Stack
• 
public class MyStageStack : Stack
• 
public class MyTestStack : Stack
...right?
g

green-stone-37839

11/06/2021, 9:16 PM
Correct, instead of that, you'd have one single 
MyStack
class with four stacks. 
Pulumi.Dev.yaml
Pulumi.Test.yaml
Pulumi.Stage.yaml
Pulumi.Prod.yaml
a

able-camera-57198

11/06/2021, 9:16 PM
Okay, so I'm on the right track there still.
I only have the one 
public class MyStack : Stack
.
g

green-stone-37839

11/06/2021, 9:17 PM
Yes.
a

able-camera-57198

11/06/2021, 9:17 PM
(FWIW, I think the name 
Stack
in code risks confusion, might be good to call that base class something like 
StackTemplate
, I might open a suggestion there 😉 )
g

green-stone-37839

11/06/2021, 9:17 PM
The .net template is misleading because it autonames it "MyStack"
yeah, exactly.
a

able-camera-57198

11/06/2021, 9:18 PM
Regardless, so coming back to the yaml files though... That 
Pulumi.yaml
does kind of result in an inference of the entrypoint.
g

green-stone-37839

11/06/2021, 9:18 PM
Please, do open an issue to rename the main class in the .net template. I think that would be helpful
💓 1
a

able-camera-57198

11/06/2021, 9:18 PM
I think because that 
Pulumi.yaml
dictates where things begin, I felt like I should have multiple 
.csproj
and multiple accompanying 
Pulumi.yaml
.
g

green-stone-37839

11/06/2021, 9:19 PM
To you 
Pulumi.yaml
file question, that represents the file Pulumi uses to understand what language SDK you're using, what the project name is, and other optional things. It has nothing to do with individual stacks
a

able-camera-57198

11/06/2021, 9:19 PM
Hmmm... Alright, so if I did this:
my-git-repo-root/
        my-stack-project-one/
            MyStackProjectOne.csproj
            Pulumi.dev.yaml
        my-stack-project-two/
            MyStackProjectTwo.csproj
            Pulumi.dev.yaml
        my-stack-project-three/
            MyStackProjectThree.csproj
            Pulumi.dev.yaml
        Pulumi.yaml
Could I get away with that?
(Of note here: Only one 
Pulumi.yaml
, above all project dirs.)
Or do the environment-specific stack 
yaml
files have to be a sibling to 
Pulumi.yaml
?
g

green-stone-37839

11/06/2021, 9:21 PM
I don't believe this will work. I think the 
Pulumi.yaml
file and the .csproj file must be in the same folder
a

able-camera-57198

11/06/2021, 9:21 PM
Okay, so it does really govern the entrypoint.
So I think that brings me back to my one issue then...
https://pulumi-community.slack.com/archives/C84L4E3N1/p1636232983433800?thread_ts=1636223171.428700&amp;cid=C84L4E3N1
I think this behaviour prevents the 
--config-file
option from working here.
g

green-stone-37839

11/06/2021, 9:23 PM
I guess I don't understand the connection. Those values would reside in the stack configuration file aka 
pulumi.dev.yaml
which has nothing to do with a 
Pulumi.yaml
file.
a

able-camera-57198

11/06/2021, 9:23 PM
Right, but I'd still be on the hook for repeating them for each 
Pulumi.yaml
I have.
So I'd have to do:
g

green-stone-37839

11/06/2021, 9:24 PM
Correct, if you want all stacks to use a single configuration file, you'd need to make use of the 
--config-file
flag.
a

able-camera-57198

11/06/2021, 9:24 PM
config:
  my-org-core:project:
  my-org-other-stack-1:project:
  my-org-other-stack-2:project:
g

green-stone-37839

11/06/2021, 9:24 PM
But that kind of defeats the purpose of a a configuration file?
why even break them up into different projects?
a

able-camera-57198

11/06/2021, 9:25 PM
Because they have a data stucture in common.
All my projects have some config that I'll need to leverage.
g

green-stone-37839

11/06/2021, 9:25 PM
If you're going to repeat the same data structure, 3 different times in the same file, why not just use 3 different stack configuration files? You're already repeating yourself 3 times.
a

able-camera-57198

11/06/2021, 9:25 PM
And they're separate projects because I'd like to be able to deploy them separately.
Yeah, I don't think --config-file is really what I want here.
What I suspect I'm after is the ability to define a snippet/base 
.yaml
and then have the stack-specific 
.yaml
files import the common values.
But then be allowed to add their own values for any specifics beyond that.
https://pulumi-community.slack.com/archives/C84L4E3N1/p1636232305430600?thread_ts=1636223171.428700&amp;cid=C84L4E3N1
Again, they may all have extra values above and beyond this.
But each one will start out with this data structure which defines a lot of values that each will use to establish conventions.
g

green-stone-37839

11/06/2021, 9:28 PM
I guess that just doesn't make sense to me. I think you're going to paint yourself into a corner. While, stack configurations, can require some duplication, I think it makes more sense to keep things siloed
You want a common config file and a stack specific file and that is not supported.
a

able-camera-57198

11/06/2021, 9:29 PM
Yarrr
It's just a pain because if I add a new region, I'm visiting one 
Pulumi.stack.yaml
for each project.
Is it the end of the world? No. I was just hoping for a nicer way to manage it.
😉
g

green-stone-37839

11/06/2021, 9:29 PM
Of course, there is nothing stopping you from writing your own. I have a python example of exactly what you're looking for: https://github.com/phillipedwards/pulumi-custom-config
a

able-camera-57198

11/06/2021, 9:30 PM
I wonder... Has anyone ever made a pulumi .net core config bridge?
g

green-stone-37839

11/06/2021, 9:31 PM
The answer is yes, but only using automation api
a

able-camera-57198

11/06/2021, 9:31 PM
Automation api?
g

green-stone-37839

11/06/2021, 9:31 PM
https://www.pulumi.com/docs/guides/automation-api/
a

able-camera-57198

11/06/2021, 9:32 PM
Ah, I think I'd sooner take duplication than try and extend a tool too much.
g

green-stone-37839

11/06/2021, 9:33 PM
How many stacks are we talking about?
5 or 50?
a

able-camera-57198

11/06/2021, 9:33 PM
6ish?
Not 50 in order of magnitude, no. More like 10 in terms of top-magnitude here 😉
g

green-stone-37839

11/06/2021, 9:33 PM
Then just duplicate them
a

able-camera-57198

11/06/2021, 9:34 PM
It's going to shake out to one project-per-git-repo which is roughly one-git-repo-per-microservice.
Each project will build on top of conventions from a single core that defines some common infrastructure. And then I'll likely also have another project for ingress/application gateway-ing/load-balancing, just to prevent any circular referencing 😉
g

green-stone-37839

11/06/2021, 9:36 PM
Are you using ComponentResources for your common infrastructure?
a

able-camera-57198

11/06/2021, 9:36 PM
Within the common infrastructure project, yeah. I have three so far. One for persistence that makes a DB and storage bucket. One for secrets. Another for some preliminary compute resources, VPC for serverless config and IAM permissions surrounding that.
I'm not sure I'll need to use component resources in my individual microservice projects though.
Probably overkill in there 😉 (but who knows, it's still early days)
g

green-stone-37839

11/06/2021, 9:37 PM
You can always refactor as you go.
Don't need to solve all the problems day one.
a

able-camera-57198

11/06/2021, 9:37 PM
Yeah, man. Pulumi is brilliant for that.
g

green-stone-37839

11/06/2021, 9:38 PM
Agreed. Its a good model to follow, IMO. It sounds like you're on the right track.
a

able-camera-57198

11/06/2021, 9:38 PM
That has to be like... The biggest thing right now is I have obliterated the pain I experienced with terraform refactoring.
Right click in Rider, click "refactor" and sing while I'm winning.
g

green-stone-37839

11/06/2021, 9:38 PM
😆 I hear you
a

able-camera-57198

11/06/2021, 9:38 PM
😄
Thanks for the guidance BTW 🙂
g

green-stone-37839

11/06/2021, 9:39 PM
anytime!
❤️ 1
a

able-camera-57198

11/06/2021, 9:40 PM
I'm sure you'll see me lurking around with deep and complex ornate questions like this for the next week or two. 😅
Just trying to avoid being an astronaut overall.
g

green-stone-37839

11/06/2021, 9:47 PM
The community is pretty rich in terms of Pulumi knowledge, so ask away. I'm out of here for the rest of the weekend. Enjoy the rest of your Saturday!
a

able-camera-57198

11/06/2021, 9:52 PM
You as well!
2 Views
#generalJoin Slack
Powered by