Are there plans to improve management of stack specific para

Question

Are there plans to improve management of stack specific parameters and secrets It seems less than half baked right now

stocky-restaurant-98004 · Answer

What are some of the features or aspects of the user experience you feel are missing

delightful-monkey-90700 · Answer

Not being infuriating to use

millions-furniture-75402 · Answer

how so

delightful-monkey-90700 · Answer

Currently it requires some yaml file locally to store the secret remotely

millions-furniture-75402 · Answer

You have a valid point about the tight coupling the Pulumi CLI has with the Stack YAMLs

delightful-monkey-90700 · Answer

Worse I have to do things with this yaml file over time

millions-furniture-75402 · Answer

What do you have to do with it over time

delightful-monkey-90700 · Answer

I want my remote secrets to be remote

millions-furniture-75402 · Answer

Have you considered not storing secrets in your stack config

delightful-monkey-90700 · Answer

I have one secret that s needed to be synchronized across cloud resources across cloud providers currently for a deployment

millions-furniture-75402 · Answer

What you can do for example in AWS Is put a secret s name in your stack config that is used to look up the secret in AWS Secrets Manager

delightful-monkey-90700 · Answer

Yes but I m using Google Cloud and AWS

delightful-monkey-90700 · Answer

In the future this secret will go away moved to Cloud HSM but for a few more months I need it

delightful-monkey-90700 · Answer

I just had to regenerate it because I renamed the stack after tearing it down and redeploying it and I guess its salted with the stack name

millions-furniture-75402 · Answer

What secrets provider do you use passphrase <https www pulumi com docs reference cli pulumi stack change secrets provider >

delightful-monkey-90700 · Answer

I m using the remote pulumi one

millions-furniture-75402 · Answer

Maybe if you used vault you wouldn t have the issue <https www pulumi com docs intro concepts secrets hashicorp vault transit secrets engine>

millions-furniture-75402 · Answer

I m not clear on whether the values for the secrets are stored encrypted locally in the config or remotely though

delightful-monkey-90700 · Answer

So is that a no on plans to improve the Pulumi one

millions-furniture-75402 · Answer

I don t know I don t work for Pulumi I just use it

stocky-restaurant-98004 · Answer

< delightful monkey 90700> We obviously want you to have a good experience using Pulumi but in order to respond to your feedback we need an actionable suggestion Can you submit a GH issue describing your use case and what you d like to see changed In the meanwhile I would try placing your secret in a centralized place AWS Secrets Manager etc if it needs to be accessed by multiple Pulumi programs keep the cloud identifier ARN etc as a config value and grab the value in your program rather than using stack config

average-tiger-58107 · Answer

This is precisely what I do Store a secret in the yaml file for the stack in the case of AWS create an SSM param whose value comes from `config requireSecret mySecret ` If you need to access that secret in other stacks perhaps ECS tasks that are consumers of that configuration value export the arn in the stack where you defined it and require the value on the consuming stack In terms of secrets across cloud providers I m not really seeing your point < delightful monkey 90700> You will always need to create resources specific to that cloud for storing secrets You may just need a better understanding of secrets management The convenience of pulumi secrets is to ensure that values remain encrypted over the network with respect to state file It is still your responsibility to choose a remote secret management tool

delightful-monkey-90700 · Answer

The point is the same secret has to be created in multiple cloud resources and with different people performing updates over time

delightful-monkey-90700 · Answer

So having a brittle local file for no good reason makes this more brittle for no good reason

average-tiger-58107 · Answer

But the secret is still defined in one place

average-tiger-58107 · Answer

You can create resources for that secret in google cloud azure and aws and then you are basically just passing around ID s across stacks

delightful-monkey-90700 · Answer

No it s defined in multiple places every single checkout has it

delightful-monkey-90700 · Answer

I want it to be defined in one place

average-tiger-58107 · Answer

Your issue is that the yaml file is checked in to version control

average-tiger-58107 · Answer

And anyone checking out the repo has access to that yaml file

delightful-monkey-90700 · Answer

I don t care that they have access to it it s just a nuisance for a remotely managed secret to also need a local file that sometimes needs to be manually changed and then everyone needs to get that

average-tiger-58107 · Answer

That local file is the source of truth for that configuration value You create a resource that is defined using that value You want to change the value change it in the yaml file run pulumi up and now all of the resources for aws gcp azure etc have updated values All the resources consuming those resources are simply pointers to its ARN

average-tiger-58107 · Answer

If you are disciplined about how updates occur to that file in practice you have a single place where you define the secret for as many clouds as you would like

delightful-monkey-90700 · Answer

Yes you have identified the problem

delightful-monkey-90700 · Answer

Do you know if there are plans to improve it

average-tiger-58107 · Answer

I don t understand how this is your problem You said you want it to be defined in one place

average-tiger-58107 · Answer

What would you like it to look like instead

delightful-monkey-90700 · Answer

And you described how it s defined in multiple places and manually synchronized and thus everyone involved must be disciplined in how we use that file for this to be successful

stocky-restaurant-98004 · Answer

I believe you can store the secret in a centralized location in Secrets Manager and then query for its value via a stable partial ID path or the full ARN at run time in your Pulumi program

average-tiger-58107 · Answer

No lol It is defined in one place

stocky-restaurant-98004 · Answer

centralized location = an AWS account used for this specific purpose or Vault etc

delightful-monkey-90700 · Answer

< average tiger 58107> And that place is

average-tiger-58107 · Answer

The YAML file

delightful-monkey-90700 · Answer

Which is located where

average-tiger-58107 · Answer

In the source of the stack where you define and update secret resources SSM Secrets Manager google cloud secrets etc Those resources consume the stack yaml file Want to update a secret change it once in the stack yaml file and run pulumi up It is now updated everywhere and any other stack referencing the arn of those resources can grab the new value without any code changes Change the YAML once get the new value everywhere This is how terraform works too

stocky-restaurant-98004 · Answer

No matter what you do you are at the very least going to have to identify where the secret can be found if not the actual value encrypted of course That can be with an ARN or similar identifier or a stack output from another stack

delightful-monkey-90700 · Answer

Sure I m fine with having a pointer to where it s located

delightful-monkey-90700 · Answer

< average tiger 58107> So if someone else changes the YAML file located on their computer not mine and does pulumi up I don t need to do anything on my computer which is again not their computer and has no shared memory or shared filesystems

stocky-restaurant-98004 · Answer

I think the easiest thing to do is to put the secret value manually in wherever you want to store secrets e g in AWS Secrets Manager an AWS account specifically for this and closely related purposes not general infra and give your Pulumi program a principal with least priv to read that secret Then you configure the Pulumi program that needs to read the secret with the ARN and then get the value at run time by querying for it

delightful-monkey-90700 · Answer

And the next time I run pulumi up on my computer which cannot read the memory or filesystem of their computer it won t affect anything as well

average-tiger-58107 · Answer

Well I would first want to think about who should have the authority to run arbitrary commands using pulumi CLI much like you don t let anybody stroll into AWS UI and update secrets by hand

average-tiger-58107 · Answer

Another tool that is really useful for avoiding conflicts across local file systems is git

stocky-restaurant-98004 · Answer

Pulumi and all IaC tools generally require full admin to the cloud provider You can mitigate this risk by running all deployments through a CI pipeline and instead grant read liberally and write perms very selectively in your org

average-tiger-58107 · Answer

The issue of two people running pulumi up locally with different code is not unique to secrets nor is it really unique to pulumi

delightful-monkey-90700 · Answer

I feel like there is nothing productive to be had with this conversation Pulumi has no plans to improve this aspect and additionally it seems like the people involved in this discussion do not understand the problem even after they correctly describe it

delightful-monkey-90700 · Answer

Bye

average-tiger-58107 · Answer

Bye Roy Keene I hope you find what you are looking for

billowy-army-68599 · Answer

While I see the original person asking the question has left the conversation this problem isn t unique to secrets it s a distributed configuration problem If you store configuration values secret or not in the `Pulumi lt stack gt yaml` then your mechanism of distribution is version control and requires human operation in order to sync those values ie git commit git push If that isn t acceptable to anyone who reads this thread there are ways around it To be clear Pulumi has no plans to be a distributed configuration engine because there are lots of those out there GCP s version is secrets manager <https cloud google com secret manager docs>

billowy-army-68599 · Answer

if you want a place to store and centralize secrets outside of stack configuration use that

billowy-army-68599 · Answer

As an aside this particular user has been extremely difficult and had an attitude problem every time they ve sauntered into this slack so I d recommend people not bother engaging with them anymore Some people just want to be rude and miserable

orange-policeman-59119 · Answer

< delightful monkey 90700> before I joined the Pulumi team I was a user of Pulumi and I did just what you re suggesting At my last role we were very early adopters and were initially skeptical of secret management Pulumi now has a SOC2 and so on We used Google Cloud KMS to encrypt our secrets and we stored them not in stack files `Pulumi yaml` but as named secrets GCP s Typescript library was used to retrieve secrets and encrypt or decrypt them The advantage of bringing your own language into Pulumi is that you can bring in outside libraries and tools I think speaking for myself and not for the team it s not a priority for us to write libraries for storing configuration in GCP AWS etc because there are already high quality libraries for that purpose usually maintained by the cloud provider themselves