https://pulumi.com logo
Powered by
c

colossal-plastic-46140

01/15/2020, 3:39 PM
Hi, in pulumi is there the concept of project level configs? Or do I have to rely on creating some kind of common stack that would hold common settings?
s

stocky-island-3676

01/15/2020, 3:45 PM
@colossal-plastic-46140 You mean general configs for multiple stacks which are in the same Pulumi project?
c

colossal-plastic-46140

01/15/2020, 3:45 PM
correct
s

stocky-island-3676

01/15/2020, 3:50 PM
@colossal-plastic-46140 Conceptually, one Pulumi project equals one code part (often in one directory). The differences between stacks are 
pulumi.Config()
inside that code which are often used for different deployment environments (
production
, 
testing
, etc.). I.e. you simply code the general configs for the project.
c

colossal-plastic-46140

01/15/2020, 3:51 PM
I was just hoping to leverage the familiar structure of pulumi.config() inside the pulumi.yaml, and was hoping there might be some built in inheritance I would be able to leverage
s

stocky-island-3676

01/15/2020, 4:03 PM
@colossal-plastic-46140 Hmm, can you give more details what you want to achieve? Inheritance between projects?
f

flat-insurance-25294

01/15/2020, 4:19 PM
@stocky-island-3676 There is no inheritance. You could always try to parse values out of `pulumi.yaml`and override with any values present in 
pulumu.<stack>.yaml
c

colossal-plastic-46140

01/15/2020, 4:21 PM
@flat-insurance-25294 Thats what I effectively want to do. But was hoping I could leverage pulumi.Config() from inside the pulumi.yaml. This would allow one to also leverage secrets.
f

flat-insurance-25294

01/15/2020, 4:23 PM
Afaik only name (via 
pulumi.getProject()
) is read from there, and I am not even sure of that, would have to check the source. I am of the opinion that it stacks should be removed as an input from outside and something defined in Pulumi since we already have a runtime (e.g Nodejs). Everything like setting configs could technically be done via env variables before writing an actual stack within pulumi.
s

stocky-island-3676

01/15/2020, 5:22 PM
You’re free to set configuration via environment. I think 
pulumi config
is intended to have the config in code, as well. That way it’s easier to share with multiple users. And you have a clear file structure for multiple stacks. @colossal-plastic-46140 To leverage secrets, you can/should use the 
config.requireSecret()
function: https://www.pulumi.com/docs/intro/concepts/config/#using-configuration-and-secrets-in-code
f

flat-insurance-25294

01/15/2020, 5:22 PM
@stocky-island-3676 How do I set config at runtime?
I mean I just want to store a value, it’s obvious that RandomInteger stores a key as a value, but I can’t get that.
s

stocky-island-3676

01/15/2020, 5:24 PM
@flat-insurance-25294 You mean you’re creating a value on first application & want to save that for future ones?
f

flat-insurance-25294

01/15/2020, 5:24 PM
I am creating a value on pulumi up inside the runtime (nodejs/typescript), yes.
s

stocky-island-3676

01/15/2020, 5:30 PM
Then you have 2 options: 1. The value is a random value. Then use https://github.com/pulumi/pulumi-random 2. Your code should always create the same value. I.e. no need to save the value outside code as config.
(I’m off for today. Please @mention me that I’m reminded to answer you tomorrow.)
f

flat-insurance-25294

01/16/2020, 2:32 PM
@stocky-island-3676 I’ve been thinking of what you said…
Your code should always create the same value. I.e. no need to save the value outside code as config.
That seems kinda strange, because you’d have scenarios where you write an initial value on deployment and reference it for updates. Correct me if my assumptions are wrong or this is considered bad practice; Say the CI feeds the initial deployment that creates AWS resources the email of the person doing the pull request. There is no way for me to save the email except using 
Random
to store the email. So I get an indirection, and worse than that, I don’t actually get access to the value the Random used, just the same result (since pulumi keeps track of state).
s

stocky-island-3676

01/16/2020, 4:38 PM
@flat-insurance-25294
you’d have scenarios where you write an initial value on deployment and reference it for updates
Yes. That scenario is quite often. E.g. to add a suffix with a random UUID. That’s what Pulumi automatically adds to resource names (as you know). I can’t follow your scenario with the e-mail address. Sounds like you want to add this as tags to the resource? What’s the use-case for it, though?
f

flat-insurance-25294

01/16/2020, 4:42 PM
Yeah tagging is a good example but not what I was thinking. How do you make sure that if you use it in a tag and create a resource with those tags, next time the tags don’t have it, will the tags be updated to remove the values its missing?
s

stocky-island-3676

01/16/2020, 4:47 PM
Yes, tags on AWS resources can be changed anytime. So will Pulumi do it, too. If some tag and/or value changed. Still, what do you want to achieve by using an e-mail address in a resource name or so?
f

flat-insurance-25294

01/16/2020, 4:48 PM
To tag it on the initial deployment, but since I can’t store it it won’t be available next time. The only option is to set it as a config on the stack creation time.
s

stocky-island-3676

01/16/2020, 4:49 PM
Sorry, didn’t got it, yet. What’s the purpose of the e-mail address on the resource?
f

flat-insurance-25294

01/16/2020, 7:53 PM
@stocky-island-3676 Say tagging the resources this pr created
@stocky-island-3676 Another example is mapping a name to an AZ. I am using RandomShuffle, obviously it stores the order it shuffles. But I can’t map a name to a particular value in its index. I could use index and modulus, but if the order of things change, then the value itself will change.
That’s a problem I am stuck with now. The ability to save values would have been good for that.
s

stocky-island-3676

01/20/2020, 11:21 AM
@flat-insurance-25294
Say tagging the resources this pr created
I see, you mean the initial PR, then. Or? BTW: I would use the link to the PR or the PR number, instead. However, you could then use the 
ResourceOption
ignoreChanges
. That way next PR’s wouldn’t overwrite it. AFAIK, you can’t restrict it to a specific tag.
f

flat-insurance-25294

01/21/2020, 1:35 AM
@stocky-island-3676 We’re using the PR number since that is the basis for all the subdomains and etc. I don’t understand how to use ``ResourceOption` `ignoreChanges`` I mean all I really need is the ability to store a value and read it with further 
pulumi up
Right now It’s hacked in with a Ruby script that creates a new stack, loops over a bunch of values and sets them as a stack config.
But I wish we could get rid of the ruby script and do everything within pulumi, INCLUDING creating a new stack
@stocky-island-3676 Another issue related to not being able to persist values, is that although We want to be in a scenario where “your code should always create the same value” We can’t if we rely on stuff like 
aws.getAvailabilityZones({ state: "available" }).names.slice(0, 3)
In an event where aws would change the response order, I would get a diff state than previously. This is a perfect place where you’d want to WRITE on initial deploy and READ on further deploys.
That being said, I am assuming 
getAvailabilityZones()
isn’t persisted behind the scenes, in that case I apologize. Edit: (as a workaround, I am sorting it myself and storing it in a RandomShuffle with a key)
Copy code
Related: <https://github.com/hashicorp/terraform/issues/11928>
s

stocky-island-3676

01/21/2020, 6:01 PM
@flat-insurance-25294
We’re using the PR number since that is the basis for all the subdomains and etc.
Interesting. Means the PR-number is part of the subdomain? That’s another scenario than tags, though.
We can’t if we rely on stuff like 
aws.getAvailabilityZones({ state: "available" }).names.slice(0, 3)
You’ve seen Terraform’s solution with using availability zone *ID*’s instead of name?: https://github.com/terraform-providers/terraform-provider-aws/issues/530#issuecomment-487238667) (found that in the Github issue what succeeds to the one you’ve posted). Does it work with Pulumi as well? If not, you should create an issue in 
pulumi-aws
. P.S.: Don’t shout too much in the channel itself, please.
f

flat-insurance-25294

01/21/2020, 6:03 PM
@stocky-island-3676 Sorry thought it was useful. Yeah so we use PR numbers as part of the subdomain for isolating different stacks (not pulimi stacks mind you) of our services. Frontend, Backend, other stuff. Regarding the Zones, I’ve defaulted to
Copy code
function getZones(upTo: number = 3): Array<string> {
    const _region = getRegion()
    const _zones = ["a", "b", "c"].map((zone) => _region + zone)
    return _zones.slice(0, upTo)
}
A lot of the AWS resources & APIs rely on the name and not the id, so the ID are not useful.
@stocky-island-3676 We keep going back and forth on this. At the end of the day, the ability to store something in the backing state is missing, and it’s necessary unless you want computations to get past values - which is dangerous The second issue, not necessary related to this, but I think Pulumi did a mistake with stacks -if I can be so blunt.
Stacks should be a runtime resource created using Typescript and the Pulimi API.
s

stocky-island-3676

01/21/2020, 6:06 PM
@flat-insurance-25294 Yes. Best is you summarize what we discussed in Github issues to get it to a long-term solution.
f

flat-insurance-25294

01/21/2020, 6:08 PM
If I can help change that, I would. Point me in the right direction - but afaik now, this is tightly coupled with your backend services, so I can’t really do anything here other than
Copy code
cp -r stack_template Pulumi.<VAR>.yaml
modify(Pulumi.<VAR>.yaml, <new_values>)
pulumi stack init <VAR>
pulumi stack select <VAR>
pulumi up --non-interactive
We also can’t rely on Pulumi for deleting resources, since things tend to fail quite often and we don’t push that new stack to our repo, only Staging and Production are permanent.
Copy code
Error deleting IAM Role (hello-world2bb21f83-5acf556): DeleteConflict: Cannot delete entity, must detach all policies first.
 error deleting S3 Bucket
There are more errors, but I guess at the end of the day, it’s just easier to find resources with tags on AWS console and delete manually .
s

stocky-island-3676

01/21/2020, 6:37 PM
• For storing the initial value in state, I would say https://github.com/pulumi/pulumi. • For the AWS resource issues, https://github.com/pulumi/pulumi-aws P.S.: I’m not a Pulumi employee. Just curious about how others experience Pulumi & to learn from it.
f

flat-insurance-25294

01/21/2020, 6:37 PM
Oh sorry, my bad!
I am very excited about pulumi. But with that comes concerns. If I were managing it, I’d skip all languages and stick to one. Preferably a statically typed one, like C# or TypeScript. I would also make a pulumi deployment self contained. No yaml files, no stacks. The 
index.ts
not only create the application, but also create the stacks needed and the accompanying resources. I also think kuberentesx wrapper is a mistake, it changes too many things which makes it hard to get help when your resource config looks too different than the yaml manifest files. In fact, one thing that I think is a low hanging fruit, is added value on iAM wrappers. The ability to create rights for resources created as part o a group. And the ability to get common rights for specific portions of the applications.
Sorry for the wall of text.
Also think stack outputs as single variables are a mistake. The idea would be that you’d export the config that holds the values. A single bag if you will containing all the necessary objects. As you see now, it’s a mess of exporting 100 variables, and then stack-referencing them. That should be builtin.
s

stocky-island-3676

01/21/2020, 6:52 PM
It seems conceptually different to what Pulumi offers. However, enough said on it. Thanks for your detailed problem description.
Powered by