https://pulumi.com logo
#general
Title
# general
l

limited-farmer-68874

12/12/2023, 6:57 PM
I was having trouble finding any information on this online: Is it at all reasonable to use a database as a source of data when building infra from pulumi? E.G. I want to build a Cloud Storage Bucket for each user organization that signs up with us, information which we store in a database table "organization". Is there a recommended way to approach this kind of situation?
r

red-match-15116

12/12/2023, 7:00 PM
Hey! You could use automation API for something like this. Your automation API server could read from the database, or be triggered on some sort of event based mechanism when the table is updated. You could then create an inline program with data passed in from the db query. There are examples here: https://github.com/pulumi/automation-api-examples Of interest might be the pulumi over http examples.
l

limited-farmer-68874

12/12/2023, 7:52 PM
Thank you! I'll take a look at this.
Is there any built-in way to still load values from a Pulumi.yaml config?
Or do I need to load the YAML file and manually set everything?
r

red-match-15116

12/12/2023, 8:53 PM
You can still access stack configuration using stack.GetConfig or stack.SetConfig. The thing to note though, is that if you're using an inline program, that program is by default executed in a temporary directory, so you either need to specify the directory you're working in (if you have a stack configuration yaml) or set all of your configuration values using
setConfig
What's your language of choice? I can point to the right examples if I know what lang you're using
The pulumi program itself will still consume configuration as it normally does.
l

limited-farmer-68874

12/12/2023, 9:03 PM
Ah, thank you. I was trying to do an inline program when what I want is a local program, just overriding some config values with things selected from our DB.
I will certainly miss the nicely formatted output on the Pulumi CLI, but it's great how easy this is. Thanks for the help.
One last question @red-match-15116, sorry! So via the automation API, is it not possible to use
set_config
to set a value to an array or object?
pulumi config set
has the option for a --path that enables this. Not seeing anything analogous for
set_config
in the Python API ..
r

red-match-15116

12/13/2023, 12:32 AM
You can do
path
with
set_config
too! I think that functionality was added not too long ago. Lemme find the link.
l

limited-farmer-68874

12/13/2023, 12:33 AM
AH.. I do see it in the source code for the lib actually! Thanks! Is there a reference/documentation for the libraries anywhere? I was trying to find it on the pulumi site but I couldn't..
r

red-match-15116

12/13/2023, 12:37 AM
Yeah, it exists but it's a little funky: https://www.pulumi.com/docs/reference/pkg/python/pulumi/#module-pulumi.automation:~:text=ConfigMap-,set_config,-(stack_name%3A There is an open issue to move this to readthedocs style python sdk documentatoin
l

limited-farmer-68874

12/13/2023, 12:39 AM
oh this is wonderful regardless, thank you!!
one day soon I will have time to contribute back to the Pulumi community. Great stuff, thank you very much for the help here
Okay -- one final question, actually! Code complete, but having some trouble getting this to run properly in a GHA, and I'm worried this is a tricky one:
Copy code
pulumi:pulumi:Stack my-org-prod  Installing dependencies in virtual environment...
    pulumi:pulumi:Stack my-org-prod  Looking in indexes: <https://pypi.org/simple>, <https://us-central1-python.pkg.dev/my-org-test-01/my-org-python-repo/simple>, <https://us-central1-python.pkg.dev/my-org-prod-01/my-org-python-repo/simple>
@ Previewing update....
    pulumi:pulumi:Stack my-org-prod  User for us-central1-python.pkg.dev: 
    pulumi:pulumi:Stack my-org-prod  error: ERROR: Exception:
    pulumi:pulumi:Stack my-org-prod 
 stderr: error: failed to discover plugin requirements: installing dependencies via '/home/runner/work/universe/universe/venv/bin/python -m pip install -r requirements.txt': exit
We have dependencies on internal python libs that we host in GCP. As a result, we need to install
keyring
and
keyrings.google-artifactregistry-auth
before running
pip install -r requirements.txt
so that
pip
knows how to find our internal Python libraries... Is there any way to get Pulumi with a local program to first install a separate
requirements-prereqs.txt
file before it tires to install the main
requirements.txt
in its virtual environment?
r

red-match-15116

12/13/2023, 4:34 PM
Heh yeah this is a bit of a tricky one. There's no automation api mechanism for this but you can always use python builtins to run shell commands at the start of your auto api script: https://www.geeksforgeeks.org/executing-shell-commands-with-python/#
l

limited-farmer-68874

12/13/2023, 4:41 PM
Unless I'm missing something, I don't see how that will help -- I already have the dependencies installed when I run the automation. But it seems with a Local Program in Pulumi it creates a new virtual subenvironment as part of its process. What I need to do is have THAT virtual environment install these additional dependencies. Is there a way I can have a shell subprocess run WITHIN the virtual env, before it tries to install
requirements.txt
?