https://pulumi.com logo
Join Slack
Powered by
Is it possible to pass Doppler secrets into a Pulu...
# general
l
Is it possible to pass Doppler secrets into a Pulumi config? For example, with 
doppler run pulumi up
and using 
${MY_VAR}
or similar in the Pulumi config?
m
I haven't used Doppler, but assuming it sets environment variables, you can reference these from within your Pulumi program using your language/runtime's support for reading from the environment -- e.g., in Node.js, 
<http://process.env.MY|process.env.MY>_VAR
.
(So you'd bypass Pulumi config altogether.)
However did you know Pulumi has its own built-in support for secrets with Pulumi ESC? This is integrated with Pulumi config as well, so you can configure a stack to use an ESC environment, and then use Pulumi to read the values. Just sharing in case you weren't aware! https://www.pulumi.com/docs/esc/
l
Ahhhh! I see!
I think what I’m trying to achieve and how I’m trying to achieve it might be conflicting in the world of Pulumi. I have two environments; 
dev
and 
prd
. Each of those environments has a corresponding Doppler stage (effectively the equivalent of 
.<http://env.dev|env.dev>
and 
.env.prd
. With that in mind, I want a way to deploy those environments without having to maintain consistency between Doppler and Pulumi config/env. I think the solution as far as I can tell would be to use a custom provider instance to the effect of;
Copy code
provider, err := scaleway.NewProvider(ctx, "pulumi-scaleway-provider", &scaleway.ProviderArgs{
	AccessKey: pulumi.String(os.Getenv("SCW_ACCESS_KEY")),
	SecretKey: pulumi.String(os.Getenv("SCW_SECRET_KEY")),
	ProjectId: pulumi.String(os.Getenv("SCW_DEFAULT_PROJECT_ID")),
})

if err != nil {
	return err
}

bucket, err := scaleway.NewObjectBucket(ctx, "my-bucket", nil , pulumi.Provider(provider))
Is that the idiomatic approach or am I bending something to fit my need rather than actually using it correctly? 😅
m
If those values are set right on the environment, you shouldn't have to do anything; the Scaleway provider should read them automatically (based on my read of the docs -- and I think I remember testing this provider once myself, so I'm pretty sure that "just works".) Out of curiosity, how many Pulumi stacks are you using? This is normally the kind of thing where you'd have two stacks, say 
Pulumi.dev.yaml
and 
Pulumi.prd.yaml
, and you'd switch between then as you switch between your Doppler "stages" using 
pulumi stack select
.
In other words, you should be able to omit that whole 
NewProvider
declaration and just start with the 
bucket
, and the Scaleway provider should automatically detect those environment variables without your having to do anything.
l
Okay, then maybe I’m doing something wrong. I do have the two stacks and had originally the two .dev and .prd config files and tried to interpolate the environment variables into them (I get why this doesn’t work).
But I’m fighting something at the moment and I can’t decide if it’s Pulumi or Scaleway. I have two projects in Scaleway and no matter what I do (even if the SCW_DEFAULT_PROJECT_ID and/or SCW_PROJECT_ID variables are set) it always deploys to the default project
As in, a Project ID that’s not specified anywhere in the configs, it seems to be falling through to my Scaleway CLI credentials and ignoring the environment variables
Bizarrely, Scaleway’s CLI providers a utility to print the current execution context with 
scw info
. If I run that outside of the Doppler stage context, I get this (some values omitted)
Copy code
Settings:
KEY                      VALUE                                             ORIGIN
config_path              /scw/config.yaml  								   default
profile                  connor-macbook                                    active_profile in config file
default_region           -                                                 default
default_zone             fr-par-1                                          default
default_organization_id  -                                                 unknown
default_project_id       -                                                 unknown
access_key               <omit>                              			   profile (connor-macbook)
secret_key               <omit>								               profile (connor-macbook)
If I run the same command in the doppler context (simply sets the SCW_* environment variables before the command is excuted)
Copy code
Settings:
KEY                      VALUE                                             ORIGIN
config_path              /scw/config.yaml  								   default
profile                  connor-macbook                                    active_profile in config file
default_region           nl-ams                                            env (SCW_DEFAULT_REGION)
default_zone             nl-ams-1                                          env (SCW_DEFAULT_ZONE)
default_organization_id  <omit>                                            env (SCW_DEFAULT_ORGANIZATION_ID)
default_project_id       <omit>                                            env (SCW_DEFAULT_PROJECT_ID)
access_key               <omit>                                            env (SCW_ACCESS_KEY)
secret_key               <omit>                                            env (SCW_SECRET_KEY)
Which is exactly what I want and these values are also correct when I create the custom pulumi provider using os.Getenv
But in any case, it always deploys to the default Scaleway project seemingly ignoring that configuration, even though it’s correct as of the line before in the Go pulumi handler
The only thing I can find that would explain why is for the Access Key and Secret Key, it has a default project set to the the default scaleway project. Although, when I’m providing the ProjectID explicitly, I would expect this to override the default set on the credentials
At least I think I’m correct to expect that, I can’t seem to null the value in Scaleway to set no preferred default project
m
Ok so even when you read these values from within the Pulumi program with 
os.Getenv
, that still doesn't apply to the project ID setting.
l
Exactly that, even though that 
os.Getenv
returns the correct project_id for a Scaleway Project that’s not the default
Let me try and just hardcode the strings into the handler and see if it makes any difference
m
Hm. Yeah it sounds like either there's a bug with the provider itself (which seems somewhat unlikely, as these look like the same settings used with the Terraform provider) or you're right that it's Scaleway itself applying this via the access key and secret key. I don't know enough about Scaleway to rule out that second one, though.
l
Just to confirm, this is in fact the correct way to pass a custom provider to a create action? Given 
provider
is the value returned from 
scaleway.NewProvider
Copy code
bucket, err := scaleway.NewObjectBucket(ctx, "my-bucket", nil, pulumi.Provider(provider))
m
I think in that case it'd just be 
bucket, err := scaleway.NewObjectBucket(ctx, "my-bucket", nil, provider)
, since you've already declared it, but I can check
Oh no, I think that's right as you have it: 
pulumi.Provider(provider)
l
Right, I tried it with the hardcoded access, secret and project id.
and it still creates it in the default project
m
Ah
l
Interestingly though, it does seem to respect the 
SCW_DEFAULT_REGION
and 
SCW_DEFAULT_ZONE
.
It just seemingly refuses to accept an override for the credentials and project id.
I’m just going to try creating something other than an Object Storage because the default project defined on the credentials relates specifically to Object Storage.
m
I wonder if you might be hitting this: https://github.com/pulumiverse/pulumi-scaleway/issues/232
l
That could absolutely be it
Because, I managed to get it to work by using my personal credentials (not an application IAM) and setting the default project on my personal credentials to the project I want to target
and it works
Although, it refuses to work with the Application IAM
Is there any way I can run a pulumi up with verbosity that would show me errors like this?
I assume if it is that issue, somewhere something is erroring?
m
l
I managed to fix this by upgrading the following dependencies. The versions installed are as of the go template used in the getting started guide. Might be worth updating those in the underlaying template as the getting started guide specifically covers a scenario that produces this issue. 👍
Copy code
require (
	<http://github.com/lbrlabs/pulumi-scaleway/sdk|github.com/lbrlabs/pulumi-scaleway/sdk> v1.11.0
	<http://github.com/pulumi/pulumi/sdk/v3|github.com/pulumi/pulumi/sdk/v3> v3.112.0
)
Oh you already did this quite recently https://github.com/pulumi/templates/pull/713/commits
I’m not sure what version between that PR and the above versions introduces the fix.
Open in Slack
PreviousNext
Powered by