https://pulumi.com logo
Powered by
Title
f

full-bear-42554

11/12/2020, 4:40 PM
Where are the 
pulumi.yaml
and 
pulumi.<stack>.yaml
schema defined?
b

broad-dog-22463

11/12/2020, 4:41 PM
https://www.pulumi.com/docs/intro/concepts/ • Stacks • Projects
f

full-bear-42554

11/12/2020, 4:47 PM
I see that https://www.pulumi.com/docs/intro/concepts/project/ documents the various configuration options for 
pulumi.yaml
.
But,https://www.pulumi.com/docs/reference/cli/pulumi_config/ doesn't seem to document the configuration options for the stack.
Those configuration options are almost definitely provider-specific, too.....
b

broad-dog-22463

11/12/2020, 4:50 PM
I'm not following what you mean here
pulumi_config is used to set either provider specific config OR application specific config
or both of course
f

full-bear-42554

11/12/2020, 5:05 PM
My mental model might be confused, so I'll rewind a bit to explain what I'm thinking.
The Pulumi binary is configured by a combination of environment variables and YAML files.
When I run 
pulumi up
in a project directory my program is executed against every single one of my stacks, which are configured in their own 
pulumi.stack.yaml
files.
b

broad-dog-22463

11/12/2020, 5:07 PM
no no
👀 1
f

full-bear-42554

11/12/2020, 5:07 PM
The project is configured using the 
pulumi.yaml
file.
b

broad-dog-22463

11/12/2020, 5:07 PM
it's executed against the stack currently selected
pulumi stack ls will show you what is currently selected
f

full-bear-42554

11/12/2020, 5:07 PM
That makes more sense. If there's no stack currently selected then....
b

broad-dog-22463

11/12/2020, 5:07 PM
it needs to be an opt in behaviour to run pulumi up on a stack
f

full-bear-42554

11/12/2020, 5:08 PM
Okay, so everything before is true, with the exception that all stacks are brought 
up
at one time.
Only the currently-selected stack is considered.
But, each 
stack
has a particular provider. It doesn't seem that the `stack`s concept abstracts over the cloud provider.
So, for example, my 
stack
(
dev
) YAML has the current state: 
config:
  gcp:project: webbie
  gcp:zone: us-west1-a
It can be inferred from this that my 
stack
is associated with a set of GCP resources, not Azure, AWs, k8ls, Tencent cloud, Baidu, etc.
What's not clear to me is what the necessary and allowable configuration parameters are for a 
stack
YAML file.
It seems like it would depend on the provider and it seems like there may be other top-level keys other than 
config
.
Where is the schema for the 
stack
YAML files defined for each provider?
b

broad-dog-22463

11/12/2020, 5:12 PM
a stack can have 100 different providers in it
the gcp:*,* azure:, aws: configs are all provider specfic - not stack specific
there is no strict schema for a stack - anything specific to pulumi (e.g. secrets managed) will be automatically managed
otherwise it's a config bag
that config may be provider specific or it may not
f

full-bear-42554

11/12/2020, 5:22 PM
Okay, that makes sense to me.
Thanks for clarifying that.
Where are the configuration options for each provider listed?
Where are the configurations options not specific to any provider listed?
I want to know which knobs I can turn and which ones I have to turn.
b

broad-dog-22463

11/12/2020, 5:24 PM
Where are the configuration options for each provider listed?
https://www.pulumi.com/docs/reference/pkg/aws/provider/ in the provider specific docs
Where are the configurations options not specific to any provider listed?
not sure what you mean here?
f

full-bear-42554

11/12/2020, 5:26 PM
General configuration options. Configuration options which are not associated with any specific provider (not an Azure config value, not a GCP config value, etc.)
b

broad-dog-22463

11/12/2020, 5:27 PM
secret managers are controlled by the CLI itself as part of a stack init https://www.pulumi.com/docs/reference/cli/pulumi_stack_init/ That adds secret provider specific configuration that isn't really user settable due to how it is calculated You can change a secret provider for a stack as follows: https://www.pulumi.com/docs/reference/cli/pulumi_stack_change-secrets-provider/
But to set a specific value as a secret, you need to use the cli command for that - so that the pulumi engine encrypts it
pulumi config set MySecretKey SecretValue --secret
f

full-bear-42554

11/12/2020, 5:28 PM
I see.
I just did 
pulumi config set hey:there true
and it was accepted.
I now have that configuration parameter in my 
dev
YAML.
So, there's no validation done on configuration keys.
b

broad-dog-22463

11/12/2020, 5:30 PM
I'm not sure what you mean by configuration keys
everything that can be done with pulumi config set is documented here - https://www.pulumi.com/docs/reference/cli/pulumi_config_get/
in the command you just ran, hey is a namespace you jsut created
f

full-bear-42554

11/12/2020, 5:33 PM
I'm struggling with how to make this clear.
Okay.
Every provider has some configuration options.
b

broad-dog-22463

11/12/2020, 5:33 PM
correct
f

full-bear-42554

11/12/2020, 5:33 PM
Where are those enumerated for each provider?
b

broad-dog-22463

11/12/2020, 5:33 PM
as part of the provider itself
f

full-bear-42554

11/12/2020, 5:33 PM
Where is the documentation for the configuration options for each provider?
b

broad-dog-22463

11/12/2020, 5:34 PM
so as part of pulumi preview / up
I just linked it above
maybe the question you are asking is, does pulumi config set validate of the key/values that are set as part of a stack config file
f

full-bear-42554

11/12/2020, 5:39 PM
That's one question. But, it seems the answer to that is "no".
I must have missed the link you pointed me to.
https://www.pulumi.com/docs/reference/pkg/aws/provider/ ?
So, https://www.pulumi.com/docs/reference/pkg/gcp/provider/ for me at GCP?
b

broad-dog-22463

11/12/2020, 5:40 PM
correct
they are provider specific
each provider has it's own provider config options doc
f

full-bear-42554

11/12/2020, 5:41 PM
And configuring these with the Provider+language source code is mutually exclusive to providing configuration options for the provider in the 
pulumi.<stack>.yaml
way?
b

broad-dog-22463

11/12/2020, 5:43 PM
not sure what you mean here?
f

full-bear-42554

11/12/2020, 5:44 PM
When I specify 
gcp:zone: us-west1-a
in 
pulumi.dev.yaml
am I "configuring the (GCP) provider"?
b

broad-dog-22463

11/12/2020, 5:45 PM
yes
f

full-bear-42554

11/12/2020, 5:45 PM
Okay.
And, when I use the SDK to create an "explicit (GCP) 
Provider
 instance ]which] may be created and passed during resource construction" am I "configuring the (GCP) provider"?
l

lemon-agent-27707

11/12/2020, 5:45 PM
This page contains info about the schema of pulumi project settings: https://www.pulumi.com/docs/intro/concepts/project/ The documentation for stack settings is a little lacking, but here's the raw Go interface that it gets parsed into (pulumi.stack.yaml); https://github.com/pulumi/pulumi/blob/46c7c327dde68ad8ad2b2cd74faccf80137fb163/sdk/go/common/workspace/project.go#L172
f

full-bear-42554

11/12/2020, 5:46 PM
That's about project settings.
My question is about provider settings.
Because, I had to set a provider value that I didn't know existed.
I'm trying to figure out where the providers' configuration requirements are documented.
b

broad-dog-22463

11/12/2020, 5:47 PM
They are documented in the provider resource docs - that’s the only place they live
f

full-bear-42554

11/12/2020, 5:47 PM
It also seems like the SDK supports configuring the provider separately from the stack. So, there is a hierarchy of stack configuration (YAML less precedence than in-source).
@broad-dog-22463 How do the configuration options for the provider resource (source control) relate to the configuration options in the stack YAML?
b

broad-dog-22463

11/12/2020, 5:48 PM
Pulumi reads them as part of Pulumi preview / Pulumi up
f

full-bear-42554

11/12/2020, 5:48 PM
Reads which ones? Both?
The YAML and source configurations of my provider(s)?
b

broad-dog-22463

11/12/2020, 5:49 PM
The yaml is one of the options for a provider - the other is environment variables for things like access keys
f

full-bear-42554

11/12/2020, 5:49 PM
...........
So I can configure with environment variables, and the YAML, and source.
And the YAML isn't checked at 
up
-time for sane configuration strings.
And the environment variables that I can configure for each provider aren't documented.(?)
Forgive me, but I'm starting to get really confused.
b

broad-dog-22463

11/12/2020, 5:51 PM
ok lets step back
f

full-bear-42554

11/12/2020, 5:51 PM
Haha, thanks.
b

broad-dog-22463

11/12/2020, 5:52 PM
for a provider, there are the following config options: 1. you can set the values as part of the pulumi stack config 2. You can use Environment variables if you so wish
I am not sure what you mean about setting them in source
You can see how the setup of a GCP provider is made in the corresponding cloud-provider page - https://www.pulumi.com/docs/intro/cloud-providers/gcp/setup/
this is the setup page only and is not the resource specific docs - the docs of the provider (and the associated resourcs) live https://www.pulumi.com/docs/reference/pkg/gcp/
regardless of using environment variables OR using the pulumi stack config file, Pulumi will try and read either - but it has a preference for stack config file specific
f

full-bear-42554

11/12/2020, 5:55 PM
I'm not sure what you mean by setting them in source
https://www.pulumi.com/docs/reference/pkg/gcp/provider/
That page is for configuring a provider using the source code of some languages (now, TS, Python, Go, and C#).
So, in addition to using environment variables, you can also use source code. You can also use a YAML file.
And, there exists some hierarchy for determining the actual configuration of a provider at 
up
-time.
b

broad-dog-22463

11/12/2020, 5:58 PM
ok, now I understand what you mean - you mean by configuring a named provider - this is an option available if you need to have multiple providers in play as part of a single pulumi application Think of an application that needs to deploy resources in 2 specific projects You can't set that at the stack level or via env vars, you would need to create a much more soped provider to allow this to happen When you create a provider this way, you need to pass that named provider to the resource itself as part of the customResourceOptions This is not the happy path that is generally used
without setting this named providers, we have a default ambient provider that is available for use. Let me give you a direct example that may help
f

full-bear-42554

11/12/2020, 5:59 PM
"soped provider"?
b

broad-dog-22463

11/12/2020, 5:59 PM
typo s/soped/scoped
f

full-bear-42554

11/12/2020, 6:00 PM
Ahhh.
b

broad-dog-22463

11/12/2020, 6:00 PM
here is a pulumi application
import * as gcp from "@pulumi/gcp";

let bucket = new gcp.storage.Bucket("test");
bucket.onObjectFinalized("Test-Finalized$", async (data, ctx) => {
    console.log("Object finalized");
    console.log("raw: " + JSON.stringify(data));
    console.log("ctx: " + JSON.stringify(ctx));
})
notice there is not provider mentioned
f

full-bear-42554

11/12/2020, 6:00 PM
Yep.
It's obtained from env-vars or the YAML.
b

broad-dog-22463

11/12/2020, 6:00 PM
this means that this application will pick up
config by either config file or Env var
f

full-bear-42554

11/12/2020, 6:00 PM
Right.
b

broad-dog-22463

11/12/2020, 6:00 PM
config file trumps env var
f

full-bear-42554

11/12/2020, 6:01 PM
You said that. That makes sense.
How do I set the "CloudAssetCustomEndpoint" in the stack file?
b

broad-dog-22463

11/12/2020, 6:02 PM
so, if we want to have a pulumi application that deploys to muliple projects / zones / regions etc then we cant use top level stack configuration to do that
f

full-bear-42554

11/12/2020, 6:02 PM
That makes sense. I understand the "multiple" provider => source, pattern.
b

broad-dog-22463

11/12/2020, 6:02 PM
we would need to do the following: 
import * as gcp from "@pulumi/gcp";

const gcpProvider = new gcp.Provider("my-named-provider", {
    project: "my-different-provider"
})

let bucket = new gcp.storage.Bucket("test");
bucket.onObjectFinalized("Test-Finalized$", async (data, ctx) => {
    console.log("Object finalized");
    console.log("raw: " + JSON.stringify(data));
    console.log("ctx: " + JSON.stringify(ctx));
}, {
    provider: gcpProvider
})
f

full-bear-42554

11/12/2020, 6:02 PM
I still don't understand how RequestTimeout maps to an 
env-var
.
b

broad-dog-22463

11/12/2020, 6:03 PM
lets say the general rule is if its a secret (e.g. a credential) then we don't suggest env vars
so requestTimeout would never really be an env var
it doesn't make sense for that to be the case
f

full-bear-42554

11/12/2020, 6:04 PM
Is that a secret?
How do I set BillingProject in the YAML?
b

broad-dog-22463

11/12/2020, 6:04 PM
well you tell me
f

full-bear-42554

11/12/2020, 6:04 PM
Hahaha
b

broad-dog-22463

11/12/2020, 6:04 PM
pulumi config set gcp:billingProject ""
pulumi config set gcp:requestTimeout
etc etc
f

full-bear-42554

11/12/2020, 6:05 PM
Ah, okay. So there is a way to associate a "provider resource property" with a YAML configuration option or an env-var.
Where is that documented?
b

broad-dog-22463

11/12/2020, 6:05 PM
every variable on this inputs doc can be set to config
https://www.pulumi.com/docs/reference/pkg/gcp/provider/#inputs
f

full-bear-42554

11/12/2020, 6:05 PM
can be set to config
That answers my question!
Where is this documented?
Or do I have to read the source code to see how it's parsed into a struct?
b

broad-dog-22463

11/12/2020, 6:06 PM
I linked it above https://www.pulumi.com/docs/intro/cloud-providers/gcp/setup/#optional-settings Notice how it's namespaced gcp:<input parameter>
f

full-bear-42554

11/12/2020, 6:07 PM
That only specifies how to configure the provider with 3 specific variables.
It doesn't specify a transformation rule from https://www.pulumi.com/docs/reference/pkg/gcp/provider/#inputs to env-var .
b

broad-dog-22463

11/12/2020, 6:08 PM
I think are thinking too much about EnvVars
f

full-bear-42554

11/12/2020, 6:08 PM
@broad-dog-22463 I did notice the namespacing. But, I didn't see that documented anywhere.
b

broad-dog-22463

11/12/2020, 6:08 PM
you asked above about the yaml right?
f

full-bear-42554

11/12/2020, 6:08 PM
My question is really about configuration, generally.
b

broad-dog-22463

11/12/2020, 6:08 PM
ok, so then here is the answer
f

full-bear-42554

11/12/2020, 6:09 PM
What are all of the ways I can configure my providers? What are the necessary parameters each provider requires? What are the optional parameters? What's the parsing hierarchy (env-var over YAML over source)?
b

broad-dog-22463

11/12/2020, 6:12 PM
1. Pulumi allows provider specific config to be set to yaml using the namespacing 
<providername>:Key
2. Pulumi allows providers to be configured in source code when not wanting to use a default provider - this means we need a specific provider block that accepts input variables The input variables in 1 and 2 are language specific e.g. 
requestTimeout
in TypeScript or 
request_timeout
in Python - the exact inputs can be found in the provider configuration docs https://www.pulumi.com/docs/reference/pkg/gcp/provider/#inputs There are certain instances e.g. for secrets (we call these out on the provider setup pages - https://www.pulumi.com/docs/intro/cloud-providers/gcp/setup/) where environment variables may be set - Environment variables are not used for every configuration option - but the ones we do use for secret values are called out on the provider setup page
the required config values for a provider are denoted with a red asterix in the provider docs
(if a value is required by the provider and doesn't have a red asterix then that is a bug in our docs)
the provider will offer an associated error message if configured incorrectly 😕
e.g 
error: 1 error occurred:
    	* missing required configuration key "aws:region": The region where AWS operations will take place. Examples are us-east-1, us-west-2, etc.
    Set a value using the command `pulumi config set aws:region <value>`.
f

full-bear-42554

11/12/2020, 6:14 PM
@broad-dog-22463 That's perfect.
Thank you!!!!!
Is what you just wrote in the docs?
I'm only one person, but, IMHO, this is what people want to know.
How do people configure stuff and what's necessary?
b

broad-dog-22463

11/12/2020, 6:15 PM
it is in the docs but maybe the issue we have is that it could be too fragmented or buried in other sections - I will talk with @brainy-church-78120 to try and see if we can surfact this much easier - we maybe need to extend the provider setup pages to point this out to make life easier for people - am more than happy to open an issue if you think that would make it more straightforward to understad
👍🏻 1
do you feel it would be better for us to make those pages much more informative about this type of thing?
f

full-bear-42554

11/12/2020, 6:20 PM
I think that have a page called 
Configuration
which serves to branch off to these pages would be really helpful.
I think even having some tooling/debugging built to help you confirm that your Configuration is sane would be really helpful.
b

broad-dog-22463

11/12/2020, 6:21 PM
gotcha! I will take that feedback on and see what we can do here to make life a bit easier when getting started
f

full-bear-42554

11/12/2020, 6:22 PM
I find (and I guess I'm not alone looking at the GitHub issues) that the 
pulumi
output on error isn't always clear enough to help one fix an issue without support.
b

broad-dog-22463

11/12/2020, 6:22 PM
I really appreciate you persisting through this to help us find the source of the confusion
f

full-bear-42554

11/12/2020, 6:22 PM
On the contrary.
Thank you, Paul. I know that was painful.
I'm sorry if I wasn't making anything clear enough.
b

broad-dog-22463

11/12/2020, 6:22 PM
Ping if you have any issues and we can get you through them!
f

full-bear-42554

11/12/2020, 6:22 PM
🙏
b

broad-dog-22463

11/12/2020, 6:23 PM
oh not at all - I think we were talking about the same things - just at different times 😄
🙇‍♂️ 1
#docsJoin Slack
Powered by