https://pulumi.com logo
Powered by
g

gorgeous-minister-41131

08/26/2021, 8:45 PM
Is there yet a way to define project-wide configuration, e.g. in the Pulumi.yaml or is that still an open req?
b

billowy-army-68599

08/26/2021, 8:47 PM
it's still currently stack specific. this issue tracks the overall ask https://github.com/pulumi/pulumi/issues/2307
g

gorgeous-minister-41131

08/26/2021, 8:49 PM
Alright.
I wonder if jsonnet is a good approach to some of this, then feed that result into a 'compiled' runtime CONFIG abstraction...
or just extend the config parser to allow multiple yaml files from the env or command line... similar to helm's --values flag
g

green-stone-37839

08/26/2021, 8:52 PM
This example is targeted at a hierarchical config, but it shows its pretty easy to create what you're looking for until the above mentioned issue is worked on. https://github.com/phillipedwards/pulumi-custom-config
g

gorgeous-minister-41131

08/26/2021, 8:52 PM
just a deep file recursive merge with a last-file wins approach would be simple enough I'd think... it's not a complete solution for allowing 
pulumi config set
command to work in the opposite direction, but I think in most cases, folks would be comfortable with it being half workable by passing multiple config files along
g

green-stone-37839

08/26/2021, 8:52 PM
Full disclosure, that's my personal repo, so take it for what it's worth.
g

gorgeous-minister-41131

08/26/2021, 8:55 PM
and now that I think about it, it would work in the other direction with 
pulumi config set --config-file myglobalproject.yml
so methinks Pulumi would just be well served to have the config parser allow multiple files passed in instead of just assuming the first detected stack one is the only one to use
or modify the parsing of Pulumi.yaml to allow for a 
config:
section...
which gets ignored for everything else except 
config.get()
stuff
b

bored-activity-40468

08/26/2021, 8:57 PM
@gorgeous-minister-41131 That's the approach I took when I started using the automation-api. Basically using whatever config file format.
💡 1
g

gorgeous-minister-41131

08/26/2021, 8:59 PM
I've always been a proponent of configuration abstraction -- I mean, if I REALLY wanted to, I could eschew all of Pulumi's built-in configuration and use soething like reClass, hiera, or whatever, and build a library since it is just code after-all, but obviously that owuldn't be ideal since it wouldn't help the idea behind the tool itself being part of that and there are numerous benefits around pulumi understanding its own configuration vs using a 3rd party or external method for getting values into pulumi.Inputs or generating pulumi Secrets [futures] by means of another library
e,g. I currently use a vault HVAC singleton for pulling vault secrets directly instead of using the transit backend,
why? because Pulumi doesn't support using a k/v <REF> in the config value, and just wants to encrypt in place. . that's fine for some usecases, but not some orgs' .. sometimes we want the secret pulled right outta vault
it's super crude, but this is what I am referring to
then I mash them together into a K8s secret, for example:
Copy code
self.add_secrets(
            items={
                "env": self.create_secret(
                    name=f"{self.default_resource_name}-env",
                    data=Output.all(
                        vault_secrets.get_instance().get_kv2_secret(
                            f"kv-{ENV_NAME}", "mysecret/1"
                        )["data"]["data"],
                        vault_secrets.get_instance().get_kv2_secret(
                            f"kv-{ENV_NAME}", "mysecret/2"
                        )["data"]["data"],
                        vault_secrets.get_instance().get_kv2_secret(
                            f"kv-{ENV_NAME}", "mysecret/3"
                        )["data"]["data"],
                    ).apply(lambda args: merge_all_output_dicts(*args)),
                    namespace=self.namespace,
                ),
            }
        )
Copy code
def merge_all_output_dicts(*all_dicts) -> Output:
    """
    Takes multiple dictionaries assembled from a pulumi Output as
    args, and merges their values into a single Output.

    This is a light-weight merge, no nested / deep-merging is performed.
    """

    return {k: v for d in all_dicts for k, v in upper_map_keys(d).items()}
anyways I was just illustrating, that in some occasions, I already have to step away from the built-in pulumi configuration because we have some requirements that some secrets are not to be stored in the actual .yaml configuration or committed to the repository, even encrypted for rotation/compliance/other purposes.
so 🤷
b

bored-activity-40468

08/26/2021, 9:07 PM
That's true, I should have been more specific. I still set config but automation-api lets you take advantage of whatever config features a language might support and handle the de-serialization that way, then set config.
g

gorgeous-minister-41131

08/26/2021, 9:07 PM
Aye yeah
I kinda got 2 different issues involved here,... the other that being the secrets provider only works with transits by default, and in some of our vault use-cases, storing the encrypted value in the yaml file in the repo just isn't an option. but it does bring up the point of, what about configs that sit outside of the scope of the project and have to be merged in?
pulumi config operates under the notion that a single (or small team) of evelopers are going to deploy a stack of things that all are related to one-another, including its configuration.
👍 1
I just think it's time for it to move and evolve beyond that is all
b

bored-activity-40468

08/26/2021, 9:09 PM
We started looking at https://github.com/mozilla/sops but I didn't get very far. Seems like a interesting way to do that.
g

gorgeous-minister-41131

08/26/2021, 9:10 PM
a previous dev where I worked at started a project called vestibule, which was supposed to be this
aka: abstract everything as a gosu port, into Env Vars
sops was going to be one of the providers 😜
b

bored-activity-40468

08/26/2021, 9:11 PM
I think I've looked at that before 🙂
g

gorgeous-minister-41131

08/26/2021, 9:11 PM
TBH we still use Vestibule for some things, it's a handy way to abstract vault away from everything
b

bored-activity-40468

08/26/2021, 9:11 PM
it sounds familiar
g

gorgeous-minister-41131

08/26/2021, 9:12 PM
but in python, hvac is easy to use so...
I ended up winding all that into the python pulumi lib
and that therein illustrates the other issue with "rolling your own config" implementation -- now I've gone off and made a language-specific implmenetion 😄
b

bored-activity-40468

08/26/2021, 9:13 PM
yeah, that makes sense
g

gorgeous-minister-41131

08/26/2021, 9:14 PM
The more and more I examine this issue the more it seems like simply allowing for thie person's suggestion makes hella sense: https://github.com/pulumi/pulumi/issues/2307#issuecomment-794209477
then I can programatically include "additional" global configs during the instantiation of Config() and not worry about the source of that file other than it matches up with the correct syntax pulumi is expecting for say, the project_name scope
👍 1
and write to it with 
pulumi config set --config-file
accordingly
I guess that name is called "the bag"
b

bored-activity-40468

08/26/2021, 9:18 PM
is this still pre-automation-api for you?
g

gorgeous-minister-41131

08/26/2021, 9:20 PM
yeah, we're just using the pulumi cli
✔️ 1
our first goal is just getting everything we were using terraform and helm/krane (for k8s) for, into pulumi first
b

bored-activity-40468

08/26/2021, 9:21 PM
yeah, understandable
2 Views
Powered by