https://pulumi.com logo
Join Slack
Powered by
Just realised it also will add the defaults set in...
# general
c
Just realised it also will add the defaults set in the Pulumi.yaml, as it takes the current value regardless if it was explicitly set in the pulumi.stack.yaml
m
Just as some food for thought, what you're doing/thinking about seems conceptually similar to creating new stacks in CI (e.g., a stack for a PR), which would typically involve getting some default values from the Pulumi.yaml paired with config values set on the fly
c
Yes, but with my defaults for most of the stacks, they remain the same, so setting them in the Pulumi.stack.yaml are the exception but a requirement.
šŸ‘ 1
I am basically making an onboarding portal for a Single tenant software Stack.
Where only a handful of things need to be set in the pulumi.stack.yaml, and the remaining is completely fine with the defaults.
m
Yes, that's quite typical I suppose.
c
There is a situation where a change in the defaults needs to occur, like a change in the software has changed the baseline resource usage, which is where I have to modify the defaults
šŸ‘ 1
And sometimes, depending on the client they need more resources than others, so we set the overrides in the pulumi.stack.yaml
šŸ‘ 1
But that model doesn't make the config editing that nice to work with when you introduce the Automation API/SDK from Pulumi
šŸ¤” 1
As it presumes that the workDir is persistent and the single point of truth (to what i can tell)
To me this looks like a complete oversight of the library.
@echoing-match-29901 @big-piano-35669, It would be great to know if this was a oversight, or is there a solution that I am just not seeing.
m
Why can't you set configuration values through the Automation API?
You initialize your stack, and then you call 
stack.set_config("my-config-key", auto.ConfigValue("my-value"))
. This is the equivalent to running 
pulumi config set my-config-key my-value
when using the CLI
Once you successfully deploy the stack, you can always run 
pulumi config refresh
(or the equivalent Automation API call) to get back the currently deployed configuration
The default values in the Pulumi.yaml are just the fallback. Once you've set a stack-specific value (whether that's in a file or "just" programmatically), it will take precedence.
c
@modern-zebra-45309 For example PULUMI.yaml has: A: "1" B: "2" C: "3" Normally my STACK file would only have the values i override so: C: "4" Fully Evaluated the Stack Config should now be: A: "1" B: "2" C: "4" But the file only has the override for "C" My issue is if I programmatically refresh the stack: Stack File has: A: "1" B: "2" C: "4" But that is a problem as if I now change my Pulumi.yaml to: A: "5" B: "2" C: "3" Now when I refresh my stack the stack file is still: A: "1" B: "2" C: "4" As it takes the last update to be pushed to the stack config in the S3, which is the last full evaluation now, which now means that it doesn't use the fallback in the Pulumi.yaml The Result I would have hoped for was that the refresh only reflected the config values explicitly set so that my Stack file would look like this: A: "5" B: "2" C: "4"
m
I see what you mean now!
c
The issue is whether I should change the fallbacks in the Pulumi.yaml, which will be the values set for the config in each stack regardless of if I modify the Pulumi.yaml
It is anything after the first deployment basically
I can understand why this is the behaviour, as i would take that
Copy code
pulumi config refresh
Doesn't expect there to be any files present as it will purely use the S3 store state of the stack
m
I think you can still solve this rather robustly: You can parse your Pulumi.yaml and set the stack config to the values to find in there prior to previewing/updating.
c
That is true but if I have modified the Pulumi.yaml and the Stack that would be a conflict
m
I've previously used AWS SSM or some secret stores to keep track of configuration and allow users to change it, but in this case, you're sending along your Pulumi.yaml anyway and it seems that you have the same defaults for every stack. As long as you don't have a situation where in some cases, a stack should not get an updated default value, reading and setting the values based on Pulumi.yaml should be sufficient.
That is true but if I have modified the Pulumi.yaml and the Stack that would be a conflict
Can you elaborate what you mean by that?
c
IF Pulumi.yaml: A: 1 stack is: A: 2 I change Pulumi.yaml to: A: 3 How do i resolve this change
Do I presume that the Pulumi.yaml is correct and override the stack
Or to i keep the Stack and ignore the default
m
It sounded like you wanted to override the value
c
But I don't was the stack config value take precedent
I would likely have to check if the old value in A in the Pulumi.yaml match the value in the stack and if so Change the Stack to the new Default
m
Well, I think for each configuration value, there can only be two options: Default takes precedence or stack takes precedence.
> I would likely have to check if the old value in A in the Pulumi.yaml match the value in the stack and if so Change the Stack to the new Default You don't even have to do a check, I think. You can just "config set key defaultValue" for all values you find in the Pulumi.yaml (or the subset you care about)
Whether that actually changes the stack configuration or not should not matter.
Unless I'm overlooking something, that would cleanly change the behavior from "stack configuration takes precedence over any default values" to "default values take precedence over stack configuration"
c
Unless it has been set in the stack config
m
No, why? You always override the stack config with the default values.
pulumi config refresh
-> loop through all default values 
pulumi config set default-key default-value
-> 
pulumi up
c
But what if the change in default value override something like requested RAM for a service
Baseline a typical client might use 600MB so that is the default
But CLIENT BIG uses 1.2GB
So if I made a change that then need the default changing to 800MB
m
You can't have it both ways: For each config value, you have to decide whether "default beats stack" or "stack beats default" is what you want
c
Then CLIENT BIG would then be revert to 800MB
But I need it to keep to 1.2GB
That is the annoying problem
m
Pulumi does "stack beats default" for all configuration values. My solution above would do "default beats stack" for all configuration values that have a default.
And you can do any mixture of the two by adding a filter: 
pulumi config refresh
-> loop through all default values that should take precedence over stack configs 
pulumi config set default-key default-value
-> 
pulumi up
c
With the Automation API as the "slack beats default" is what I want, but as I have to do 'pulumi config refresh' it now would make it a "default beats stack"
m
No, it would not, right? You first refresh the config, then you override the config, and then you do your operations
c
When i refresh the config, it would have the old default + changed values
āœ… 1
So if i change the default
How would i know what the changed values are
Which means I would have to do "default beats stack"
m
You don't have to know that. Let me make an example, one second.
c
As the stack now has all values
Actually you know what IDK if changing the defaults will now change the stack when config refreshes
I'll also test this
I might have been running in circles for no reason
Can confirm changing the defaults in the pulumi.yaml doesn't work
When refreshing the config
m
Copy code
# Pulumi.yaml version 1:

config:
  valueA: default1
  valueB: default2

pulumi stack init my-stack

pulumi set config valueC custom-value

pulumi up

pulumi config refresh

# Pulumi.my-stack.yaml version 1:

config:
  my-stack:valueA: default1
  my-stack:valueB: default2
  my-stack:valueC: custom-value

# Pulumi.yaml version 2

config:
  valueA: default1
  valueB: new-default  # <-- you make an update

# Loop and update
pulumi config set valueA default1     # <-- no change, doesn't matter
pulumi config set valueB new-default  # <-- default takes precedence

Pulumi.my-stack.yaml version 2

config:
  my-stack:valueA: default1
  my-stack:valueB: new-value
  my-stack:valueC: custom-value
When you have default values where you only want to update if they are still set to their previous default value you'll need a bit more logic, and I don't think there could be a one-size-fits-all solution
c
Yeah that i what I mean
m
I think it would be conceivable to have a 
pulumi config reset-to-defaults
but for "only fall back to defaults when its still set to default" you would need to keep track of all previous default values
c
I need to know the old default to check if the actually need to set the config to the new default
šŸ‘ 1
That is my problem tho
How do i know the old default
I don't have tracking of the config
This is the hard part
For me atleast
m
Yes, and this is also a gnarly problem.
c
Without making this extremely complicated
for what i need it to be
Unless I make my own config tracking and mass applying of values with in the application I am making
m
For example, say that you first had "5GB" as the default and now "2GB" is the default. But now you set one of your heavier workloads to "5GB". Would you then reset this to "2GB" because "5GB" previously was a default?
c
That is a another edge case for this
m
I think instead of solving this through your default configuration, solve it through your programs.
c
I would take the presumption that the 5GB is now 2GB under our logic
Having a Mass set Value makes more sense
And then set the defaults
for new stacks
It is manual work but it is less manual work
m
Don't use a default configuration (at least not in Pulumi.yaml, you can still read the default values from a YAML or JSON file to track them in one place) but set the values like this:
Copy code
myResource('foo', {parameter: config.get('my-config-key') || DEFAULTS['my-config-key']})
Then you either get your custom override, or the current default.
c
That won't work as config after the first deploy will always be value
m
I think this has exactly the behavior you want: If you have overridden the default value by setting 'my-config-key', you will always keep that custom value. If you have not set a config value, the stack will always use the current default for that value.
c
Yeah
Exactly
m
So by default, you don't set any of the configuration values. You just have an empty stack config.
You only set the values that you want to override.
c
But unless I track the config outside of pulumi.stack.yaml files I'll never know the values I actually set
Because the Refresh of Config will just mix it.
m
Two ideas here: 1) Make a 
default.yaml
that has the same schema that the `Pulumi.yaml`'s 
config
block uses that you read all defaults from and 2) export the default values as stack outputs so you can inspect them
Because the Refresh of Config will just mix it.
The config refresh will only affect 1) values that do not have a default and 2) values that you have overridden the default for
c
Oh then I know the defaults that were used for that specific deployment
šŸ‘ 1
m
You keep that file in Git alongside the Pulumi.yaml
c
Thats actually super smart.
I might not need the default.yaml if i just parse the Pulumi.yaml tho right?
m
When you have a default config in the Pulumi.yaml, 
config.get('my-key')
will never not be set
c
Because There will still be the diff from the stack outputs and the Pulumi.yaml config
m
This does not help you, because then you run into the "how do I track whether I've overridden a default value" again
I think the key is to think about the Pulumi.yaml "config" as "this is the template for when I make a new stack or my stack does not have a particular value set in its config yet"
What you want is a dynamic fallback for when a configuration value is not set
Or, said differently, I believe what we've worked out is that under no circumstances should a default value find its way into the stack configuration. You only want default-overrides or values that never get a default in there
c
Okay, I got a plan now
šŸ‘ 1
m
Using a separate source for the default values and 
myResource('foo', {parameter: config.get('my-config-key') || DEFAULTS['my-config-key']})
gives you: ā€¢ 'my-config-key' is not set in the stack configuration -> use whichever is the current default value ā€¢ 'my-config-key' is set in the stack configuration -> always use this value, no matter the current default
30 Views
Open in Slack
PreviousNext
Powered by