https://pulumi.com logo
Title
a

astonishing-monitor-96741

06/29/2021, 12:50 PM
Morning everyone. Is it possible to determine if a value is a secret at runtime? I'm using a structured configuration and eventually looping through all the <JsonElement>s and pushing these to AWS Parameter Store. Everything works, other than when looping through I don't know which values are secrets or not, as I'd like to push them to the parameter store as "SecureString" if possible. I'm using C# if that comes in to play.
b

brave-planet-10645

06/29/2021, 12:53 PM
Yes. There is an
isSecret
method you can use. Let me just grab the link
a

astonishing-monitor-96741

06/29/2021, 12:54 PM
Awesome. I honestly did try searching before asking, I promise.
b

brave-planet-10645

06/29/2021, 12:56 PM
https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/pulumi/#isSecret for nodejs... I think it's something like
.isSecretAsync()
for .net
a

astonishing-monitor-96741

06/29/2021, 12:57 PM
Thanks!
b

bored-oyster-3147

06/29/2021, 1:57 PM
btw for .net it is static on
Output.IsSecretAsync(...)
🌟 1
a

astonishing-monitor-96741

06/29/2021, 3:44 PM
@bored-oyster-3147 @brave-planet-10645 Ok, I'm struggling a bit. Since I'm pulling this out of the config as a JsonElement all my values are JsonElements/strings.
JsonElement data = config.RequireObject<JsonElement>("Settings");
Looping through the JsonElements I want, if I Console.WriteLine, it knows its a secret and outputs accordingly.
Console.WriteLine(JsonSerializer.Serialize(data, new JsonSerializerOptions { WriteIndented = true }));

{
        "Region": "us-east-1",
        "SecretKey": "[secret]",
        "User": "service-worker"
}
But I'm struggling to use the Output.IsSecretAsync passing in a string or JsonElement. I tried creating a new Pulumi.Output.Create with the value, but still returns False.
var val = Pulumi.Output.Create(item.Value);
                var isSecret = Task.Run(async () => await Output.IsSecretAsync(val));
                var isSecret2 = await Output.IsSecretAsync(val);                
                Console.WriteLine($"IsSecret: {isSecret.Result}");
                Console.WriteLine($"IsSecret: {isSecret2}");
I'm noobing something up here but I've gone cross eyed.
b

bored-oyster-3147

06/29/2021, 3:45 PM
oh my bad I goofed. I don't know if
Output.IsSecretAsync
is correct for this because this is coming from config so isn't an output?
a

astonishing-monitor-96741

06/29/2021, 3:45 PM
Correct
b

bored-oyster-3147

06/29/2021, 3:47 PM
Don't you need to know if it's a secret ahead of time? Like it's the difference between using
config.RequireObject<T>
and
config.RequireSecretObject<T>
?
a

astonishing-monitor-96741

06/29/2021, 3:48 PM
I'm pulling in a whole constructed config chunk as:
config.RequireObject<JsonElement>("Settings");
Looping over the tree
config:
  Application:Settings:
    AppSettings:
      AllowedLoginAttempts: 4
      CaptchaInfo:
        CaptchaKey: FAKEKEY
        CaptchaSecret:
          secure: REMOVED
These are just getting piped into the parameter store for use by the application that Pulumi is spinning up
b

bored-oyster-3147

06/29/2021, 3:49 PM
But when you do that I don't think pulumi is aware of anything inside that object. You're just getting the raw
JsonElement
I think. You might need to manually check if the property has the
{ "secret": { ... } }
shape inside the
JsonElement
Sorry I mean the
secure
property
a

astonishing-monitor-96741

06/29/2021, 3:50 PM
When I loop through the tree and Console.Writeline it definitely prints everything but the secrets. It prints them as [secret]
b

bored-oyster-3147

06/29/2021, 3:51 PM
yes that's because you did
RequireObject
which pulumi is not using your passphrase to deserialize so it just has the plaintext
[secret]
for those values
a

astonishing-monitor-96741

06/29/2021, 3:51 PM
It pushes the correct value to the parameter store though, the unecrypted value of the secret
b

bored-oyster-3147

06/29/2021, 3:52 PM
what is the type at the time that you pass it to the parameter store?
a

astonishing-monitor-96741

06/29/2021, 3:54 PM
Name = name,
 Value = item.Value.ToString()
JsonElement.Value.ToString()
b

bored-oyster-3147

06/29/2021, 3:54 PM
Then I confess that at the moment I have no idea how that is functioning. I would have to do some digging
a

astonishing-monitor-96741

06/29/2021, 3:54 PM
I piped it into an Stack Output variable too after I created it as a new Output<string> and it had the decrypted value there as well
Should I take that as you are going to do some digging? Or should I make a more formal request with support? Thanks for the help/communication thus far.
b

bored-oyster-3147

06/29/2021, 4:04 PM
So I'm honestly not sure how the
JsonElement.Value.ToString()
is doing any kind of decryption but I guess that's not super relevant to your issue. To get back to your issue, it looks like internally the
Deployment
instance keeps a hash set of all of the fully qualified configuration keys that are secrets and uses this method to check that collection. It doesn't appear that this is exposed any where that I have been able to find, so maybe we could open an issue to expose something like that on the
Config
object
a

astonishing-monitor-96741

06/29/2021, 4:10 PM
Is that something you'd like me to do?
b

bored-oyster-3147

06/29/2021, 4:12 PM
I'm just a contributor 🙂 @brave-planet-10645 might have another idea but that is what I would do
a

astonishing-monitor-96741

06/29/2021, 4:12 PM
Gotcha
b

brave-planet-10645

06/29/2021, 4:12 PM
Let me catch up with the thread
👍 1
a

astonishing-monitor-96741

06/29/2021, 4:22 PM
Going to step out to lunch, but I'll check back
b

brave-planet-10645

06/29/2021, 5:09 PM
So JsonElement is in
System.Text.Json
so Pulumi is just converting the secret config value and passing through as
[secret]
let me ask internally here. I think we're missing something in our docs
a

astonishing-monitor-96741

06/29/2021, 5:35 PM
Ya, because it definitely knows its a secret specifically somehow. I get the correct value as stackoutput and in AWS, and its
[secret]
like you mention if I write it out. Below is the full code I'm using to pull it out of the config, flatten the hierarchy, and write it to console. Later on I loop through the settings collection to add each key/value to AWS parameter store. Everything works perfect, other than I'd like to recognize the secrets and add them to parameter store as SecureString.
public AppConfig()
        {
            var config = new Pulumi.Config("Application");
            JsonElement data = config.RequireObject<JsonElement>("Settings");  
            
            SettingsCollection = GetFlat(data.ToString());            

            Console.WriteLine("*************************************");
            Console.WriteLine(JsonSerializer.Serialize(SettingsCollection, new JsonSerializerOptions { WriteIndented = true }));
            Console.WriteLine("*************************************");          
        }

        public static Dictionary<string, JsonElement> GetFlat(string? json)
        {
            if (json == null) return new Dictionary<string, JsonElement>();

            IEnumerable<(string Path, JsonProperty P)> GetLeaves(string? path, JsonProperty p)
                => p.Value.ValueKind != JsonValueKind.Object
                    ? new[] { (Path: path == null ? p.Name : path + "/" + p.Name, p) }
                    : p.Value.EnumerateObject().SelectMany(child => GetLeaves(path == null ? p.Name : path + "/" + p.Name, child));

            using (JsonDocument document = JsonDocument.Parse(json)) 
                return document.RootElement.EnumerateObject()
                    .SelectMany(p => GetLeaves(null, p))
                    .ToDictionary(k => k.Path, v => v.P.Value.Clone()); //Clone so that we can use the values outside of using
        }
Example config with pulumi CLI created structured secret
config:
  Application:Settings:
    AppSettings:
      AllowedLoginAttempts: 4
      CaptchaInfo:
        CaptchaKey: FAKEKEYasdfasdfasdf
        CaptchaSecret:
          secure: REMOVEDojiijfjio32ji23jfj0f3093j923j09
Code for the looping and ssm creation:
foreach (var item in appConfig.SettingsCollection)
            {
                string name = $"/{prefix}/{item.Key}";
                new Ssm.Parameter(name, new Ssm.ParameterArgs
                {
                    Name = name,
                    Value = item.Value.ToString() ?? "[ERROR] config entry null",
                    Type = "String",
                    DataType = "text"
                });
            }
b

brave-planet-10645

06/29/2021, 6:18 PM
So you need to do
config.RequireSecretObject()
b

bored-oyster-3147

06/29/2021, 6:23 PM
I'm still confused about how his use case is getting the decrypted values at all. I would expect it to be plaintext
[secret]
especially after seeing his
GetFlat(...)
function since he's just getting that nested
JsonElement
and doing
JsonElement.Value.ToString()
So it seems like he just needs to do
RequireSecretObject<JsonElement>()
but not so that his pulumi program functions (cause it does currently) just so that the secrets are decrypted early for the purpose of his string usage
But if he does
RequireSecretObject<JsonElement>
he still has no way of knowing which of those nested elements are actually secrets so that he can conditionally use
SecureString
in his SSM parameter resource
a

astonishing-monitor-96741

06/29/2021, 6:25 PM
Ya, I wasn't sure how that was going to change anything either. : /
@brave-planet-10645 Did they have any insight on how the RequireSecretObject would help? Once I do the Apply in order to loop through the object I still don;'t know which individual values were secret.
b

brave-planet-10645

06/29/2021, 6:54 PM
No you won't get that information. It decrypts them all. So you have two choices here. You could either split the config model up into two and have "secret" and a "not-secret" sections, or raise an feature request here and describe what you're looking for
a

astonishing-monitor-96741

06/29/2021, 6:56 PM
Roger.
b

brave-planet-10645

06/29/2021, 8:34 PM
Thank you