https://pulumi.com logo
Join Slack
Powered by
This message was deleted.
# docs
s
This message was deleted.
b
f
I see that https://www.pulumi.com/docs/intro/concepts/project/ documents the various configuration options for 
pulumi.yaml
.
But,https://www.pulumi.com/docs/reference/cli/pulumi_config/ doesn't seem to document the configuration options for the stack.
Those configuration options are almost definitely provider-specific, too.....
b
I'm not following what you mean here
pulumi_config is used to set either provider specific config OR application specific config
or both of course
f
My mental model might be confused, so I'll rewind a bit to explain what I'm thinking.
The Pulumi binary is configured by a combination of environment variables and YAML files.
When I run 
pulumi up
in a project directory my program is executed against every single one of my stacks, which are configured in their own 
pulumi.stack.yaml
files.
b
no no
👀 1
f
The project is configured using the 
pulumi.yaml
file.
b
it's executed against the stack currently selected
pulumi stack ls will show you what is currently selected
f
That makes more sense. If there's no stack currently selected then....
b
it needs to be an opt in behaviour to run pulumi up on a stack
f
Okay, so everything before is true, with the exception that all stacks are brought 
up
at one time.
Only the currently-selected stack is considered.
But, each 
stack
has a particular provider. It doesn't seem that the `stack`s concept abstracts over the cloud provider.
So, for example, my 
stack
(
dev
) YAML has the current state:
Copy code
config:
  gcp:project: webbie
  gcp:zone: us-west1-a
It can be inferred from this that my 
stack
is associated with a set of GCP resources, not Azure, AWs, k8ls, Tencent cloud, Baidu, etc.
What's not clear to me is what the necessary and allowable configuration parameters are for a 
stack
YAML file.
It seems like it would depend on the provider and it seems like there may be other top-level keys other than 
config
.
Where is the schema for the 
stack
YAML files defined for each provider?
b
a stack can have 100 different providers in it
the gcp:*,* azure:, aws: configs are all provider specfic - not stack specific
there is no strict schema for a stack - anything specific to pulumi (e.g. secrets managed) will be automatically managed
otherwise it's a config bag
that config may be provider specific or it may not
f
Okay, that makes sense to me.
Thanks for clarifying that.
Where are the configuration options for each provider listed?
Where are the configurations options not specific to any provider listed?
I want to know which knobs I can turn and which ones I have to turn.
b
Where are the configuration options for each provider listed?
https://www.pulumi.com/docs/reference/pkg/aws/provider/ in the provider specific docs
Where are the configurations options not specific to any provider listed?
not sure what you mean here?
f
General configuration options. Configuration options which are not associated with any specific provider (not an Azure config value, not a GCP config value, etc.)
b
secret managers are controlled by the CLI itself as part of a stack init https://www.pulumi.com/docs/reference/cli/pulumi_stack_init/ That adds secret provider specific configuration that isn't really user settable due to how it is calculated You can change a secret provider for a stack as follows: https://www.pulumi.com/docs/reference/cli/pulumi_stack_change-secrets-provider/
But to set a specific value as a secret, you need to use the cli command for that - so that the pulumi engine encrypts it
pulumi config set MySecretKey SecretValue --secret
f
I see.
I just did 
pulumi config set hey:there true
and it was accepted.
I now have that configuration parameter in my 
dev
YAML.
So, there's no validation done on configuration keys.
b
I'm not sure what you mean by configuration keys
everything that can be done with pulumi config set is documented here - https://www.pulumi.com/docs/reference/cli/pulumi_config_get/
in the command you just ran, hey is a namespace you jsut created
f
I'm struggling with how to make this clear.
Okay.
Every provider has some configuration options.
b
correct
f
Where are those enumerated for each provider?
b
as part of the provider itself
f
Where is the documentation for the configuration options for each provider?
b
so as part of pulumi preview / up
I just linked it above
maybe the question you are asking is, does pulumi config set validate of the key/values that are set as part of a stack config file
f
That's one question. But, it seems the answer to that is "no".
I must have missed the link you pointed me to.
b
correct
they are provider specific
each provider has it's own provider config options doc
f
And configuring these with the Provider+language source code is mutually exclusive to providing configuration options for the provider in the 
pulumi.<stack>.yaml
way?
b
not sure what you mean here?
f
When I specify 
gcp:zone: us-west1-a
in 
pulumi.dev.yaml
am I "configuring the (GCP) provider"?
b
yes
f
Okay.
And, when I use the SDK to create an "explicit (GCP) 
Provider
 instance ]which] may be created and passed during resource construction" am I "configuring the (GCP) provider"?
l
This page contains info about the schema of pulumi project settings: https://www.pulumi.com/docs/intro/concepts/project/ The documentation for stack settings is a little lacking, but here's the raw Go interface that it gets parsed into (pulumi.stack.yaml); https://github.com/pulumi/pulumi/blob/46c7c327dde68ad8ad2b2cd74faccf80137fb163/sdk/go/common/workspace/project.go#L172
f
That's about project settings.
My question is about provider settings.
Because, I had to set a provider value that I didn't know existed.
I'm trying to figure out where the providers' configuration requirements are documented.
b
They are documented in the provider resource docs - that’s the only place they live
f
It also seems like the SDK supports configuring the provider separately from the stack. So, there is a hierarchy of stack configuration (YAML less precedence than in-source).
@broad-dog-22463 How do the configuration options for the provider resource (source control) relate to the configuration options in the stack YAML?
b
Pulumi reads them as part of Pulumi preview / Pulumi up
f
Reads which ones? Both?
The YAML and source configurations of my provider(s)?
b
The yaml is one of the options for a provider - the other is environment variables for things like access keys
f
...........
So I can configure with environment variables, and the YAML, and source.
And the YAML isn't checked at 
up
-time for sane configuration strings.
And the environment variables that I can configure for each provider aren't documented.(?)
Forgive me, but I'm starting to get really confused.
b
ok lets step back
f
Haha, thanks.
b
for a provider, there are the following config options: 1. you can set the values as part of the pulumi stack config 2. You can use Environment variables if you so wish
I am not sure what you mean about setting them in source
You can see how the setup of a GCP provider is made in the corresponding cloud-provider page - https://www.pulumi.com/docs/intro/cloud-providers/gcp/setup/
this is the setup page only and is not the resource specific docs - the docs of the provider (and the associated resourcs) live https://www.pulumi.com/docs/reference/pkg/gcp/
regardless of using environment variables OR using the pulumi stack config file, Pulumi will try and read either - but it has a preference for stack config file specific
f
I'm not sure what you mean by setting them in source
https://www.pulumi.com/docs/reference/pkg/gcp/provider/
That page is for configuring a provider using the source code of some languages (now, TS, Python, Go, and C#).
So, in addition to using environment variables, you can also use source code. You can also use a YAML file.
And, there exists some hierarchy for determining the actual configuration of a provider at 
up
-time.
b
ok, now I understand what you mean - you mean by configuring a named provider - this is an option available if you need to have multiple providers in play as part of a single pulumi application Think of an application that needs to deploy resources in 2 specific projects You can't set that at the stack level or via env vars, you would need to create a much more soped provider to allow this to happen When you create a provider this way, you need to pass that named provider to the resource itself as part of the customResourceOptions This is not the happy path that is generally used
without setting this named providers, we have a default ambient provider that is available for use. Let me give you a direct example that may help
f
"soped provider"?
b
typo s/soped/scoped
f
Ahhh.
b
here is a pulumi application
Copy code
import * as gcp from "@pulumi/gcp";

let bucket = new gcp.storage.Bucket("test");
bucket.onObjectFinalized("Test-Finalized$", async (data, ctx) => {
    console.log("Object finalized");
    console.log("raw: " + JSON.stringify(data));
    console.log("ctx: " + JSON.stringify(ctx));
})
notice there is not provider mentioned
f
Yep.
It's obtained from env-vars or the YAML.
b
this means that this application will pick up
config by either config file or Env var
f
Right.
b
config file trumps env var
f
You said that. That makes sense.
How do I set the "CloudAssetCustomEndpoint" in the stack file?
b
so, if we want to have a pulumi application that deploys to muliple projects / zones / regions etc then we cant use top level stack configuration to do that
f
That makes sense. I understand the "multiple" provider => source, pattern.
b
we would need to do the following:
Copy code
import * as gcp from "@pulumi/gcp";

const gcpProvider = new gcp.Provider("my-named-provider", {
    project: "my-different-provider"
})

let bucket = new gcp.storage.Bucket("test");
bucket.onObjectFinalized("Test-Finalized$", async (data, ctx) => {
    console.log("Object finalized");
    console.log("raw: " + JSON.stringify(data));
    console.log("ctx: " + JSON.stringify(ctx));
}, {
    provider: gcpProvider
})
f
I still don't understand how RequestTimeout maps to an 
env-var
.
b
lets say the general rule is if its a secret (e.g. a credential) then we don't suggest env vars
so requestTimeout would never really be an env var
it doesn't make sense for that to be the case
f
Is that a secret?
How do I set BillingProject in the YAML?
b
well you tell me
f
Hahaha
b
pulumi config set gcp:billingProject ""
pulumi config set gcp:requestTimeout
etc etc
f
Ah, okay. So there is a way to associate a "provider resource property" with a YAML configuration option or an env-var.
Where is that documented?
b
every variable on this inputs doc can be set to config
f
can be set to config
That answers my question!
Where is this documented?
Or do I have to read the source code to see how it's parsed into a struct?
b
I linked it above https://www.pulumi.com/docs/intro/cloud-providers/gcp/setup/#optional-settings Notice how it's namespaced gcp:<input parameter>
f
That only specifies how to configure the provider with 3 specific variables.
It doesn't specify a transformation rule from https://www.pulumi.com/docs/reference/pkg/gcp/provider/#inputs to env-var .
b
I think are thinking too much about EnvVars
f
@broad-dog-22463 I did notice the namespacing. But, I didn't see that documented anywhere.
b
you asked above about the yaml right?
f
My question is really about configuration, generally.
b
ok, so then here is the answer
f
What are all of the ways I can configure my providers? What are the necessary parameters each provider requires? What are the optional parameters? What's the parsing hierarchy (env-var over YAML over source)?
b
1. Pulumi allows provider specific config to be set to yaml using the namespacing 
<providername>:Key
2. Pulumi allows providers to be configured in source code when not wanting to use a default provider - this means we need a specific provider block that accepts input variables The input variables in 1 and 2 are language specific e.g. 
requestTimeout
in TypeScript or 
request_timeout
in Python - the exact inputs can be found in the provider configuration docs https://www.pulumi.com/docs/reference/pkg/gcp/provider/#inputs There are certain instances e.g. for secrets (we call these out on the provider setup pages - https://www.pulumi.com/docs/intro/cloud-providers/gcp/setup/) where environment variables may be set - Environment variables are not used for every configuration option - but the ones we do use for secret values are called out on the provider setup page
the required config values for a provider are denoted with a red asterix in the provider docs
(if a value is required by the provider and doesn't have a red asterix then that is a bug in our docs)
the provider will offer an associated error message if configured incorrectly 😕
e.g
Copy code
error: 1 error occurred:
    	* missing required configuration key "aws:region": The region where AWS operations will take place. Examples are us-east-1, us-west-2, etc.
    Set a value using the command `pulumi config set aws:region <value>`.
f
@broad-dog-22463 That's perfect.
Thank you!!!!!
Is what you just wrote in the docs?
I'm only one person, but, IMHO, this is what people want to know.
How do people configure stuff and what's necessary?
b
it is in the docs but maybe the issue we have is that it could be too fragmented or buried in other sections - I will talk with @brainy-church-78120 to try and see if we can surfact this much easier - we maybe need to extend the provider setup pages to point this out to make life easier for people - am more than happy to open an issue if you think that would make it more straightforward to understad
👍🏻 1
do you feel it would be better for us to make those pages much more informative about this type of thing?
f
I think that have a page called 
Configuration
which serves to branch off to these pages would be really helpful.
I think even having some tooling/debugging built to help you confirm that your Configuration is sane would be really helpful.
b
gotcha! I will take that feedback on and see what we can do here to make life a bit easier when getting started
f
I find (and I guess I'm not alone looking at the GitHub issues) that the 
pulumi
output on error isn't always clear enough to help one fix an issue without support.
b
I really appreciate you persisting through this to help us find the source of the confusion
f
On the contrary.
Thank you, Paul. I know that was painful.
I'm sorry if I wasn't making anything clear enough.
b
Ping if you have any issues and we can get you through them!
f
🙏
b
oh not at all - I think we were talking about the same things - just at different times 😄
🙇‍♂️ 1
Open in Slack
PreviousNext
Powered by