would really value your input on

Question

< red match 15116> would really value your input on < full artist 27215> amp < purple appointment 84502> s feedback above Looks like this has been raised before as far back as 2018 even by Joe himself see but has yet to be resolved Is there any mechanism allowing pulumi to be invoked from or for automation s `work dir` param to be set to a higher level directory than the project dir e g a directory further up the hierarchy from the one containing the yaml and main py files That way common code can be shared between multiple pulumi projects as sibling or parent modules packages and imported in main py and python would resolve them as they d naturally be included in sys path Or is there a different more obvious solution or workaround that I m missing

red-match-15116 · Answer

Does this not work <https github com pulumi pulumi issues 1641 issuecomment 453263416>

enough-leather-70274 · Answer

Sure that works but that imported module is within the project dir itself

enough-leather-70274 · Answer

The problem is trying to import a module or package that s shared across multiple pulumi projects in a monorepo In this case you want them defined somewhere else in the tree and to run python from a common ancestor

enough-leather-70274 · Answer

Our structure is what < purple appointment 84502> specified

red-match-15116 · Answer

Ah got it Thanks for clarifying

red-match-15116 · Answer

So your resources are defined in components

enough-leather-70274 · Answer

where deploy main py deploys au and us infrastructure projects via pulumi automation We `auto select stack` passing in work dir as infra us

enough-leather-70274 · Answer

Yeah infra us main py has `from infra components import Vpc`

red-match-15116 · Answer

What doesn t work

enough-leather-70274 · Answer

`ModuleNotFoundError No module named infra `

red-match-15116 · Answer

This happens when you run `python main py` from deploy

enough-leather-70274 · Answer

Same same

enough-leather-70274 · Answer

it s because only infra us is on sys path

red-match-15116 · Answer

same to what What two things result in that error Sorry just trying to wrap my head around this

enough-leather-70274 · Answer

yeah sorry

enough-leather-70274 · Answer

same for both the CLI and auto

enough-leather-70274 · Answer

CLI is obvious because you have cd to infra us to run pulumi up

red-match-15116 · Answer

Ah okay got it

enough-leather-70274 · Answer

but running the automation script in deploy results in the same presumably because setting the work dir to infra us in effect does the same as using the CLI in that directory

red-match-15116 · Answer

gt presumably because setting the work dir to infra us in effect does the same as using the CLI in that directory Yeah that s right

enough-leather-70274 · Answer

Other python tools I ve worked with usually decouple configuration from the execution context

enough-leather-70274 · Answer

allowing you to pass in a path via a param or a cli arg to the config file s

enough-leather-70274 · Answer

and then you can invoke that tool from wherever you want in our case that s really the infra directory

red-match-15116 · Answer

What happens if you do `from components import Vpc` I hear you I m just not sure we have the capacity to take this on right now so I m trying to figure out if there s a workaround

enough-leather-70274 · Answer

Yup I ll settle for a workaround too slightly smiling face

enough-leather-70274 · Answer

Sadly relative imports don t work either Error is the same for both CLI and automation `ImportError attempted relative import with no known parent package`

enough-leather-70274 · Answer

note that same error occurs whether I make the pulumi project a package or not

red-match-15116 · Answer

Ugh bummer

red-match-15116 · Answer

gt Other python tools I ve worked with usually decouple configuration from the execution context I think there s a way to do this Gimme a second

red-match-15116 · Answer

Hmm okay no I m coming up short unfortunately I guess the only hack that seems viable is to manipulate `sys path` as < full artist 27215> mentioned

enough-leather-70274 · Answer

Yeah I ve not had luck with that so far either disappointed

red-match-15116 · Answer

gt Yeah I ve not had luck with that so far either Hmm how do you mean I would think ```sys path append os path abspath components py ``` would then let you do `from components import Vpc`

enough-leather-70274 · Answer

Yup my bad I was appending an existing pathlib handle which I thought contained the parent but didn t

enough-leather-70274 · Answer

Strategically what would be the best solution to this

enough-leather-70274 · Answer

Allowing the CLI to be invoked from outside the project dir maybe passing in config via an arg which itself references where main py exists and then modifying automation to allow the same

enough-leather-70274 · Answer

Happy to write whatever the best mechanism would be as a feature request

red-match-15116 · Answer

I m honestly not sure what the solution is yet but just capturing the details of what you described in this thread would be super helpful so we can bring it up for prioritization

enough-leather-70274 · Answer

Will do Thanks for your help as always

red-match-15116 · Answer

So there actually is a mechanism to indicate where your ` main py` file is by passing in a `main` key to the `Pulumi yaml` file but I m not sure that actually solves your problem since you have multiple pulumi projects in one repo

enough-leather-70274 · Answer

Indeed This is more about python being able to resolve module imports outside the project directory and it doesn t seem like pulumi be it cli or automation enables that right now

red-match-15116 · Answer

Okay another idea you can install local packages into your `venv` with `pip install e` I m not sure if that would work with your directory structure since `infra` contains both the components and the pulumi programs so dunno if that would cause some weird recursive isses but if there was a sibling directory to `infra` called `resources` or something which contained `components py` and ` init py` you could conceivably `pip install e path to resources` and then be able to do `from resources components import Vpc` Again not the ideal solution but a possible workaround

red-match-15116 · Answer

Also noticed this <https www pulumi com docs intro languages python ~ text=although 20the 20template 20uses 20a 20very 20simple 20package 20structure 2C 20by 20placing 20 main py 20in 20the 20root 20directory 2C 20pulumi 20fully 20supports 20properly 20modularized 20python 20programs 20and 20setup py 20files 20this 20is 20important 20if 20you 20ever 20decide 20to 20turn 20your 20pulumi 20program 20into 20a 20library |Note> in the docs

enough-leather-70274 · Answer

Yeah I ll admit I m not sure what that note means no detail on how to package a project that isn t defined in the root dir

enough-leather-70274 · Answer

Using pip for a local package is a nice idea tho I ll give that a try

full-artist-27215 · Answer

What I ended up settling on at the moment is having my projects in sibling directories to my shared code In each project directory I have my ` main py` file along with `Pulumi yaml` and my stack config files All my shared code imports are absolute and I add ```import sys sys path insert 0 ``` to the top of each ` main py` file So far this works with a minimal amount of fuss though being able to control things more concretely with some CLI arguments would ultimately be fantastic I just `cd pulumi lt PROJECT gt ` and run `pulumi up` and things work I m maybe doing some unconventional virtualenv management though since all the projects share a virtualenv that I manage outside of Pulumi s built in support via project config options That may affect the overall suitability of this approach somewhat

full-artist-27215 · Answer

happy to share more details if that s useful

enough-leather-70274 · Answer

Cheers < full artist 27215> that s exactly where I ended up with < red match 15116> too

full-artist-27215 · Answer

I m by no means a Python expert but maybe tweaking Pulumi to be able to run a module via `runpy run module` <https docs python org 3 library runpy html runpy run module> as opposed to just `runpy run path` would be useful for dealing with this kind of usecase I think that might alleviate the need to manually patch `sys path`

enough-leather-70274 · Answer

FWIW Komal your local package suggestion does work but seems to require a setup py Until pulumi offers a better solution I think I prefer the sys path hack + a comment as it s more explicit and lives right alongside the problematic imports

great-sunset-355 · Answer

The CLI accepts CWD string I wonder if it would help in your case ``` C cwd string Run pulumi as if it had been started in another directory```

great-sunset-355 · Answer

I ll soon be dividing my code into reusable modules as well

enough-leather-70274 · Answer

Nice find < great sunset 355> I d missed that option Quick check sadly it seems to change the execution context of python to that directory

enough-leather-70274 · Answer

In my example above I cd to infra and run `pulumi cwd=us preview` I get `ModuleNotFoundError No module named infra `

enough-leather-70274 · Answer

If I cd to us and run `pulumi cwd= preview` I get `error no Pulumi yaml project file found`

great-sunset-355 · Answer

ouch well expect to deal with the same similar problem around next week time sorry I cannot spend more time on this now But it would be great if you could share a repo with minimal example of your setup

brave-knife-93369 · Answer

As a datapoint we are hitting the same issue We have a pulumi stack embedded into a project as a subdirectory and no amount of fiddling with sys path seems to make all use cases happy unit tests running pulumi from the subdirectory manually and calling the stack from the automation API If stacks are limited to a main file everything works but as soon as we break things out into separate files things blow up If there is any way we can influence the roadmap that s our 1 blocker for wider pulumi adoption confused

red-match-15116 · Answer

< brave knife 93369> I see that you saw < billowy army 68599> s reference to the mit project that has a fairly complex project structure in the linked thread If this continues to be a 1 blocker I would suggest starting with opening an issue and we will prioritize accordingly but it does appear there is existing prior art that you can base your project off of

brave-knife-93369 · Answer

Yes I think this will allow us not to have to resort to path hacks I ll report back on how well that works for us Thanks

enough-leather-70274 · Answer

FYI < red match 15116> I ve opened and replicated documented the issue at

red-match-15116 · Answer

Thanks < enough leather 70274> Curious if you ve looked at the mit example and tried to replicate their workflow it seems to work for them

enough-leather-70274 · Answer

Yeah as < brave knife 93369> pointed out it requires packaging the repo and installing in a venv Not sure that s a great fit for our use case we re not re using or publishing the projects themselves but guess it s an alternate workaround to hacking the syspath Once again the latter feels clearer as to what we re working around and why it s there i e maybe just better for us from a maintenance PoV