https://pulumi.com logo
Join Slack
Powered by
Are there plans to improve management of stack-spe...
# general
d
Are there plans to improve management of stack-specific parameters and secrets ? It seems less than half-baked right now.
s
What are some of the features or aspects of the user experience you feel are missing?
d
Not being infuriating to use ?
m
how so?
d
Currently it requires some .yaml file locally... to store the secret remotely...
m
You have a valid point about the tight coupling the Pulumi CLI has with the Stack YAMLs
d
Worse, I have to do things with this yaml file over time
m
What do you have to do with it over time?
d
I want my remote secrets to be remote
m
Have you considered not storing secrets in your stack config?
d
I have one secret that's needed to be synchronized across cloud resources across cloud providers currently for a deployment
m
What you can do, for example in AWS. Is put a secret's name in your stack config, that is used to look up the secret in AWS Secrets Manager.
d
Yes, but I'm using Google Cloud and AWS
In the future this secret will go away (moved to Cloud HSM), but for a few more months I need it
I just had to regenerate it because I renamed the stack (after tearing it down and redeploying it) and I guess its salted with the stack name
m
d
I'm using the remote pulumi one
m
I'm not clear on whether the values for the secrets are stored encrypted locally in the config or remotely though
d
So is that a "no" on plans to improve the Pulumi one ?
m
I don't know, I don't work for Pulumi, I just use it.
✅ 1
s
@delightful-monkey-90700 We obviously want you to have a good experience using Pulumi, but in order to respond to your feedback, we need an actionable suggestion. Can you submit a GH issue describing your use case and what you'd like to see changed? https://github.com/pulumi/pulumi/issues In the meanwhile, I would try placing your secret in a centralized place (AWS Secrets Manager, etc.) if it needs to be accessed by multiple Pulumi programs, keep the cloud identifier (ARN, etc.) as a config value and grab the value in your program rather than using stack config.
a
This is precisely what I do. Store a secret in the yaml file for the stack, in the case of AWS, create an SSM param whose value comes from 
config.requireSecret("mySecret")
. If you need to access that secret in other stacks (perhaps ECS tasks that are consumers of that configuration value), export the arn in the stack where you defined it and require the value on the consuming stack. In terms of secrets across cloud providers, I'm not really seeing your point @delightful-monkey-90700. You will always need to create resources specific to that cloud for storing secrets. You may just need a better understanding of secrets management. The convenience of pulumi secrets is to ensure that values remain encrypted over the network with respect to state file. It is still your responsibility to choose a "remote" secret management tool.
d
The point is the same secret has to be created in multiple cloud resources, and with different people performing updates over time
So having a brittle local file for no good reason makes this more brittle for no good reason
a
But the secret is still defined in one place
You can create resources for that secret in google cloud, azure and aws, and then you are basically just passing around ID's across stacks
d
No, it's defined in multiple places -- every single checkout has it
I want it to be defined in one place
a
Your issue is that the yaml file is checked in to version control?
And anyone checking out the repo has access to that yaml file?
d
I don't care that they have access to it -- it's just a nuisance for a remotely managed secret to also need a local file that sometimes needs to be manually changed and then everyone needs to get that
a
That local file is the source of truth for that configuration value. You create a resource that is defined using that value. You want to change the value? change it in the yaml file, run pulumi up, and now all of the resources for aws, gcp, azure, etc have updated values. All the resources consuming those resources are simply pointers to its ARN
If you are disciplined about how updates occur to that file, in practice you have a single place where you define the secret for as many clouds as you would like
d
Yes, you have identified the problem.
Do you know if there are plans to improve it ?
a
I don't understand how this is your problem. You said you want it to be defined in one place
What would you like it to look like instead?
d
And you described how it's defined in multiple places and manually synchronized and thus everyone involved must be disciplined in how we use that file for this to be successful
s
I believe you can store the secret in a centralized location in Secrets Manager and then query for its value via a stable partial ID (path) or the full ARN at run time in your Pulumi program.
a
No lol - It is defined in one place
s
"centralized location" = "an AWS account used for this specific purpose" or Vault, etc.
d
@average-tiger-58107 And that place is ?
a
The YAML file
d
Which is located where ?
a
In the source of the stack where you define and update secret resources (SSM, Secrets Manager, google cloud secrets, etc). Those resources consume the stack yaml file. Want to update a secret, change it once, in the stack yaml file, and run pulumi up. It is now updated everywhere, and any other stack referencing the arn of those resources can grab the new value without any code changes. Change the YAML once, get the new value everywhere. This is how terraform works too
s
No matter what you do, you are, at the very least, going to have to identify where the secret can be found if not the actual value (encrypted, of course). That can be with an ARN or similar identifier, or a stack output from another stack.
d
Sure, I'm fine with having a pointer to where it's located
@average-tiger-58107 So if someone else changes the YAML file (located on their computer, not mine) and does "pulumi up", I don't need to do anything on my computer (which is, again, not their computer and has no shared memory or shared filesystems) ?
s
I think the easiest thing to do is to put the secret value manually in wherever you want to store secrets (e.g. in AWS Secrets Manager an AWS account specifically for this and closely related purposes, not general infra) and give your Pulumi program a principal with least priv to read that secret. Then you configure the Pulumi program that needs to read the secret with the ARN, and then get the value at run time by querying for it.
d
And the next time I run "pulumi up" on my computer (which cannot read the memory or filesystem of their computer), it won't affect anything as well ?
a
Well, I would first want to think about who should have the authority to run arbitrary commands using pulumi CLI, much like you don't let anybody stroll into AWS UI and update secrets by hand.
Another tool that is really useful for avoiding conflicts across local file systems is git
s
Pulumi (and all IaC tools) generally require full admin to the cloud provider. You can mitigate this risk by running all deployments through a CI pipeline and instead grant read liberally and write perms very selectively in your org.
✅ 2
a
The issue of two people running pulumi up locally with different code is not unique to secrets, nor is it really unique to pulumi
d
I feel like there is nothing productive to be had with this conversation. Pulumi has no plans to improve this aspect and additionally it seems like the people involved in this discussion do not understand the problem even after they correctly describe it.
Bye.
a
Bye Roy Keene. I hope you find what you are looking for
b
While I see the original person asking the question has “left” the conversation, this problem isn’t unique to secrets, it’s a distributed configuration problem. If you store configuration values (secret or not) in the 
Pulumi.<stack>.yaml
then your mechanism of distribution is version control and requires human operation in order to sync those values (ie, git commit/git push) If that isn’t acceptable to anyone who reads this thread, there are ways around it. To be clear, Pulumi has no plans to be a distributed configuration engine because there are lots of those out there GCP’s version is secrets manager: https://cloud.google.com/secret-manager/docs
🙏 1
if you want a place to store and centralize secrets outside of stack configuration, use that.
As an aside, this particular user has been extremely difficult and had an attitude problem every time they’ve sauntered into this slack, so I’d recommend people not bother engaging with them anymore. Some people just want to be rude and miserable
✅ 6
o
@delightful-monkey-90700 before I joined the Pulumi team, I was a user of Pulumi and I did just what you're suggesting. (At my last role we were very early adopters and were initially skeptical of secret management. Pulumi now has a SOC2 and so on.) We used Google Cloud KMS to encrypt our secrets and we stored them not in stack files (
Pulumi.*.yaml
) but as named secrets. GCP's Typescript library was used to retrieve secrets and encrypt or decrypt them. The advantage of bringing your own language into Pulumi is that you can bring in outside libraries and tools. I think - speaking for myself and not for the team - it's not a priority for us to write libraries for storing configuration in GCP, AWS, etc because there are already high quality libraries for that purpose, usually maintained by the cloud provider themselves.
5 Views
Open in Slack
PreviousNext
Powered by