Channels
pulumi-ai
package-authoring
pulumi-cdk
oracle-cloud-infrastructure
learn-pulumi-events
linen
registry
built-with-pulumi
pulumi-cloud
contribex
testingtesting321
hacktoberfest
cloudengineering
yaml
pulumi-crosscode
content-share
finops
multi-language-hackathon
office-hours
workshops
blog-posts
localstack
welcome
gitlab
pulumi-kubernetes-operator
jobs
aws
pulumi-deployments
dotnet
golang
announcements
java
pulumiverse
python
install
getting-started
cloudengineering-support
testingtesting123
hackathon-03-19-2020
typescript
google-cloud
contribute
azure
kubernetes
docs
automation-api
status
general
Title
d
delightful-monkey-90700
09/16/2022, 8:25 PMAre there plans to improve management of stack-specific parameters and secrets ? It seems less than half-baked right now.
s
stocky-restaurant-98004
09/16/2022, 8:30 PMWhat are some of the features or aspects of the user experience you feel are missing?
d
delightful-monkey-90700
09/16/2022, 8:30 PMNot being infuriating to use ?
m
millions-furniture-75402
09/16/2022, 8:31 PMhow so?
d
delightful-monkey-90700
09/16/2022, 8:31 PMCurrently it requires some .yaml file locally... to store the secret remotely...
m
millions-furniture-75402
09/16/2022, 8:31 PMYou have a valid point about the tight coupling the Pulumi CLI has with the Stack YAMLs
d
delightful-monkey-90700
09/16/2022, 8:31 PMWorse, I have to do things with this yaml file over time
m
millions-furniture-75402
09/16/2022, 8:31 PMWhat do you have to do with it over time?
d
delightful-monkey-90700
09/16/2022, 8:32 PMI want my remote secrets to be remote
m
millions-furniture-75402
09/16/2022, 8:32 PMHave you considered not storing secrets in your stack config?
d
delightful-monkey-90700
09/16/2022, 8:32 PMI have one secret that's needed to be synchronized across cloud resources across cloud providers currently for a deployment
m
millions-furniture-75402
09/16/2022, 8:32 PMWhat you can do, for example in AWS. Is put a secret's name in your stack config, that is used to look up the secret in AWS Secrets Manager.
d
delightful-monkey-90700
09/16/2022, 8:33 PMYes, but I'm using Google Cloud and AWS
In the future this secret will go away (moved to Cloud HSM), but for a few more months I need it
I just had to regenerate it because I renamed the stack (after tearing it down and redeploying it) and I guess its salted with the stack name
m
millions-furniture-75402
09/16/2022, 8:34 PMWhat secrets provider do you use? passphrase?
https://www.pulumi.com/docs/reference/cli/pulumi_stack_change-secrets-provider/
d
delightful-monkey-90700
09/16/2022, 8:37 PMI'm using the remote pulumi one
m
millions-furniture-75402
09/16/2022, 8:39 PMMaybe if you used vault you wouldn't have the issue? https://www.pulumi.com/docs/intro/concepts/secrets/#hashicorp-vault-transit-secrets-engine
I'm not clear on whether the values for the secrets are stored encrypted locally in the config or remotely though
d
delightful-monkey-90700
09/16/2022, 8:40 PMSo is that a "no" on plans to improve the Pulumi one ?
m
millions-furniture-75402
09/16/2022, 8:41 PMI don't know, I don't work for Pulumi, I just use it.
✅ 1
s
stocky-restaurant-98004
09/16/2022, 8:46 PM@delightful-monkey-90700 We obviously want you to have a good experience using Pulumi, but in order to respond to your feedback, we need an actionable suggestion. Can you submit a GH issue describing your use case and what you'd like to see changed? https://github.com/pulumi/pulumi/issues
In the meanwhile, I would try placing your secret in a centralized place (AWS Secrets Manager, etc.) if it needs to be accessed by multiple Pulumi programs, keep the cloud identifier (ARN, etc.) as a config value and grab the value in your program rather than using stack config.
a
average-tiger-58107
09/16/2022, 9:43 PMThis is precisely what I do. Store a secret in the yaml file for the stack, in the case of AWS, create an SSM param whose value comes from
config.requireSecret("mySecret")d
delightful-monkey-90700
09/16/2022, 9:45 PMThe point is the same secret has to be created in multiple cloud resources, and with different people performing updates over time
So having a brittle local file for no good reason makes this more brittle for no good reason
a
average-tiger-58107
09/16/2022, 9:46 PMBut the secret is still defined in one place
You can create resources for that secret in google cloud, azure and aws, and then you are basically just passing around ID's across stacks
d
delightful-monkey-90700
09/16/2022, 9:48 PMNo, it's defined in multiple places -- every single checkout has it
I want it to be defined in one place
a
average-tiger-58107
09/16/2022, 9:49 PMYour issue is that the yaml file is checked in to version control?
And anyone checking out the repo has access to that yaml file?
d
delightful-monkey-90700
09/16/2022, 9:51 PMI don't care that they have access to it -- it's just a nuisance for a remotely managed secret to also need a local file that sometimes needs to be manually changed and then everyone needs to get that
a
average-tiger-58107
09/16/2022, 9:53 PMThat local file is the source of truth for that configuration value. You create a resource that is defined using that value. You want to change the value? change it in the yaml file, run pulumi up, and now all of the resources for aws, gcp, azure, etc have updated values. All the resources consuming those resources are simply pointers to its ARN
If you are disciplined about how updates occur to that file, in practice you have a single place where you define the secret for as many clouds as you would like
d
delightful-monkey-90700
09/16/2022, 9:55 PMYes, you have identified the problem.
Do you know if there are plans to improve it ?
a
average-tiger-58107
09/16/2022, 9:56 PMI don't understand how this is your problem. You said you want it to be defined in one place
What would you like it to look like instead?
d
delightful-monkey-90700
09/16/2022, 9:57 PMAnd you described how it's defined in multiple places and manually synchronized and thus everyone involved must be disciplined in how we use that file for this to be successful
s
stocky-restaurant-98004
09/16/2022, 10:00 PMI believe you can store the secret in a centralized location in Secrets Manager and then query for its value via a stable partial ID (path) or the full ARN at run time in your Pulumi program.
a
average-tiger-58107
09/16/2022, 10:01 PMNo lol - It is defined in one place
s
stocky-restaurant-98004
09/16/2022, 10:01 PM"centralized location" = "an AWS account used for this specific purpose" or Vault, etc.
d
delightful-monkey-90700
09/16/2022, 10:05 PM@average-tiger-58107 And that place is ?
a
average-tiger-58107
09/16/2022, 10:08 PMThe YAML file
d
delightful-monkey-90700
09/16/2022, 10:09 PMWhich is located where ?
a
average-tiger-58107
09/16/2022, 10:15 PMIn the source of the stack where you define and update secret resources (SSM, Secrets Manager, google cloud secrets, etc). Those resources consume the stack yaml file. Want to update a secret, change it once, in the stack yaml file, and run pulumi up. It is now updated everywhere, and any other stack referencing the arn of those resources can grab the new value without any code changes. Change the YAML once, get the new value everywhere. This is how terraform works too
s
stocky-restaurant-98004
09/16/2022, 10:16 PMNo matter what you do, you are, at the very least, going to have to identify where the secret can be found if not the actual value (encrypted, of course). That can be with an ARN or similar identifier, or a stack output from another stack.
d
delightful-monkey-90700
09/16/2022, 10:17 PMSure, I'm fine with having a pointer to where it's located
@average-tiger-58107 So if someone else changes the YAML file (located on their computer, not mine) and does "pulumi up", I don't need to do anything on my computer (which is, again, not their computer and has no shared memory or shared filesystems) ?
s
stocky-restaurant-98004
09/16/2022, 10:19 PMI think the easiest thing to do is to put the secret value manually in wherever you want to store secrets (e.g. in AWS Secrets Manager an AWS account specifically for this and closely related purposes, not general infra) and give your Pulumi program a principal with least priv to read that secret. Then you configure the Pulumi program that needs to read the secret with the ARN, and then get the value at run time by querying for it.
d
delightful-monkey-90700
09/16/2022, 10:19 PMAnd the next time I run "pulumi up" on my computer (which cannot read the memory or filesystem of their computer), it won't affect anything as well ?
a
average-tiger-58107
09/16/2022, 10:21 PMWell, I would first want to think about who should have the authority to run arbitrary commands using pulumi CLI, much like you don't let anybody stroll into AWS UI and update secrets by hand.
Another tool that is really useful for avoiding conflicts across local file systems is git
s
stocky-restaurant-98004
09/16/2022, 10:23 PMPulumi (and all IaC tools) generally require full admin to the cloud provider. You can mitigate this risk by running all deployments through a CI pipeline and instead grant read liberally and write perms very selectively in your org.
✅ 2
a
average-tiger-58107
09/16/2022, 10:23 PMThe issue of two people running pulumi up locally with different code is not unique to secrets, nor is it really unique to pulumi
d
delightful-monkey-90700
09/16/2022, 10:25 PMI feel like there is nothing productive to be had with this conversation. Pulumi has no plans to improve this aspect and additionally it seems like the people involved in this discussion do not understand the problem even after they correctly describe it.
Bye.
a
average-tiger-58107
09/16/2022, 10:27 PMBye Roy Keene. I hope you find what you are looking for
b
billowy-army-68599
09/17/2022, 1:45 AMWhile I see the original person asking the question has “left” the conversation, this problem isn’t unique to secrets, it’s a distributed configuration problem.
If you store configuration values (secret or not) in the
Pulumi.<stack>.yaml🙏 1
if you want a place to store and centralize secrets outside of stack configuration, use that.
As an aside, this particular user has been extremely difficult and had an attitude problem every time they’ve sauntered into this slack, so I’d recommend people not bother engaging with them anymore. Some people just want to be rude and miserable
✅ 6
o
orange-policeman-59119
10/03/2022, 9:35 PM@delightful-monkey-90700 before I joined the Pulumi team, I was a user of Pulumi and I did just what you're suggesting. (At my last role we were very early adopters and were initially skeptical of secret management. Pulumi now has a SOC2 and so on.)
We used Google Cloud KMS to encrypt our secrets and we stored them not in stack files (
Pulumi.*.yaml