The web console (in Activity / Changes, for the “D...
# general
c
The web console (in Activity / Changes, for the “Diff” and also the file obtained via “Down stack checkpoint”) shows the raw output of each state change ,which including things like plaintext passwords, even for things marked as secrets. The “Resources” view correctly identifies these as encrypted (as per https://www.pulumi.com/docs/intro/concepts/secrets/)
1: What is the story re: encryption of these state changes / activity logs on Pulumi servers?
2: Is it possible to opt-in to using Pulumi cloud to hold the state, but to opt out of Pulumi cloud holding the activity log? So for example, when I create a AWS
SecretVersion
and then run
pulumi up
on my local machine, I expect the secret to created directly in the AWS Secrets Manager. The secret is also stored in Pulumi state (encrypted as per the document above) and also in the activity / diff log available from the web console (which appears to be in plaintext)
As an experiment, I created a new project with a custom
secretsprovider
(using a key in AWS KMS) and then created a new
Secret
/
SecretVersion
. As before, the activity section of the Pulumi web console show the secret in plaintext, clearly indicating that the secrets are being transferred to the Pulumi web console backend in plaintext (over TLS of course, but that isn’t the point)
By enabling tracing (
pulumi -v 999 up --logtostderr
) this can clearly be seen in the JSON payload uploaded to Pulumi (i.e. calls to `https://api.pulumi.com/api/stacks/fuji/secrettest/dev/update/xxx/checkpoint`) contains the resource for the
SecretVersion
which includes the
inputs
which contains the
secretString
plaintext:
Copy code
"inputs": {
  "secretId": "arn:aws:secretsmanager:...",
  "secretString": "xxx1234yyy"
}
It appears that the only way to avoid “leaking” secrets to the Pulumi cloud backend is to avoid using the cloud backend entirely and use either self-managed (which comes with many caveats) or self-hosted (enterprise only, and removes a key benefit of using Pulumi in the first place)
Can someone more knowledgeable comment on whether there is a way to use the Pulumi managed service, but avoiding the above security pitfall?
h
i think you should be able to use pulumi config set --secret to store your secrets as ciphertext. didn't look at the thread so not sure if you have already tried this.
r
If you pass the input to
Secret
as a pulumi secret it will not be stored in plaintext. So like:
Copy code
const cfg = new pulumi.Config()
const param = new aws.ssm.Parameter("a-secret-param", {
    type: "SecureString",
    value: cfg.requireSecret("my-secret-value"),
});
Will never be stored as plaintext in the checkpoint file. You can also use your own secret provider and the service will never have access to the decrypted secret.
c
Thanks you @red-match-15116 - I’m aware of Pulumi secret config and also using an external service for performing the encryption (indeed, the example above is doing the latter). This isn’t quite what i’m looking for however. I seem to be doing a very poor job explaining what I’m asking! Let me try again
What the above solution says is, there is a way to store secrets in Pulumi config in encrypted form and, better yet, this can be done using an encryption key unavailable to Pulumi - and that’s great
And if you do this, none of your plaintext secrets are ever sent to Pulumi in checkpoints - again, great
What I want is to avoid placing my secrets in Pulumi at all, regardless of whether Pulumi is able to decrypt them or not
As it stands, it seems that this is not possible - which is my real question, is that true?
r
What I want is to avoid placing my secrets in Pulumi at all, regardless of whether Pulumi is able to decrypt them or not
Yeah, that's not possible unless you use self-managed or self-hosted as you described above.
c
Thanks for confirming @red-match-15116. I’ll raise a feature request for this, as it would be sad to miss out on the benefits of using the Pulumi managed backend due to this. Rationale: Creating an encrypted/secret config entry to avoid plaintext secrets from being sent to the backend (as part of the checkpoint data) is an undesirable workaround when you consider that the Pulumi backend doesn’t actually need the secret at all. A better solution is to not send the secrets to the Pulumi backend at all, minimize the attack surface/audit surface. Also, the secrets themselves may be rotated (i.e. auto key rotation on AWS secretmanager) and so the encrypted secret that Pulumi holds in state will be stale anyway and so not even useful for “has this changed?” purposes (I tested this out to confirm the behaviour). The feature request is therefore something like (BDD style): “as a devops user, I want to use the Pulumi managed service for managing my cloud infra state without sharing any plaintext or encrypted secret values that are not required to be held by the service to manage that state”
f
Could you provide an example of the request above? In your example using an SSM Parameter, the actual secret value is “part of the state” in that if you modify the underlying secret value, then pulumi should detect the diff and update that value. You could choose not to manage that part of the state via pulumi and purposely have it drift from state via
ignoreChanges
and feeding in an initial dummy value that you later change out of band of Pulumi.
Regarding the rotation example where things change in the provider, state can become stale due to out of changes, but a
pulumi refresh
would read the value and update the underlying state.
r
Rationale: Creating an encrypted/secret config entry to avoid plaintext secrets from being sent to the backend (as part of the checkpoint data) is an undesirable workaround when you consider that the Pulumi backend doesn’t actually need the secret at all.
The Pulumi backend does need the secret if you are creating the secret in Pulumi because it is part of the `Secret`'s state. It also needs the secret if you are passing it as an input into another resource (because it is part of that resource's state). If you are using Pulumi to manage your state, it seems inconceivable that it wouldn't need your secret in some form - since that secret is inherently part of the state that you are asking Pulumi to manage.
c
I wasn’t aware of
ignoreChanges
@faint-table-42725, thank you for flagging that, it is certainly useful here.
The design I’ve arrived at is (hopefully also makes the use case clear): • Create an RDS
Instance
with an ephemeral bootstrap password (lets call it
p1
) with
ignoreChanges
on
password
• Create a
Secret
(
s1
) and
SecretVersion
with a
secret_string
that contains
p1
with
ignoreChanges
on
secret_string
• Create a
SecretRotation
for
s1
(and associated Lambda function) • Create an application which reads the RDS password from
s1
So I make no attempt to hide the initial RDS password from Pulumi state or checkpoints as it is (nearly) immediately rotated on creation
As you point out, a
pulumi refresh
would update the password (is that still true when
ignoreChanges
is used? I haven’t tried it seems so). Is this operation ever invoked as part of another command or only ever explicitly?
I did consider also using a Pulumi secret config for the (short lived) bootstrap password
p1
but that felt like overkill
Would you be kind enough to critique approach I’ve outlined?
cc @red-match-15116
r
Is this operation ever invoked as part of another command or only ever explicitly?
Refresh is only ever called explicitly or by setting a Pulumi.yaml config setting to always refresh on update. Your approach sounds reasonable to me!
c
Thank you both very much, your help has been invaluable!