https://pulumi.com logo
Title
m

millions-pharmacist-626

12/22/2022, 7:01 PM
Hey everyone, I'm building a repo that contains multiple pulumi projects and I ran into the same issue described here and here, that is to say, you can't import in
__main__.py
modules from parent directories. Mostly the solutions that have been suggested are: ā€¢ Either hack sys.path in some way or another, ā€¢ Install the pulumi repo itself as a package as described in this thread I was wondering if in the meantime, someone had come up with a preferrable solution
c

clever-painter-96148

12/22/2022, 7:21 PM
My pulumi programs are very minimalistic, and I put all my logic in a library package. So I can use standard python tooling and relative imports in the actual package without having to hack around stuff.
m

millions-pharmacist-626

12/22/2022, 7:22 PM
I put all my logic in a library package.
So you create a separate folder, like
logic
that sits next to your projects, and add that as a dependency ?
c

clever-painter-96148

12/22/2022, 7:23 PM
Yes!
m

millions-pharmacist-626

12/22/2022, 7:23 PM
that's an interesting alternative
what would be the pros and cons of doing what you recommend vs. installing the project as a local package with either
poetry install
or
pip install .
?
c

clever-painter-96148

12/22/2022, 7:25 PM
When you wrote "add that as a dependency" I understood "poetry add"
so I do not understand your last question?
m

millions-pharmacist-626

12/22/2022, 7:25 PM
nono it's exactly what I meant.
poetry add ./mylocalpackage
c

clever-painter-96148

12/22/2022, 7:27 PM
In `project-gke-cluster/pyproject.toml`:
[tool.poetry.dependencies]
python = "^3.10"
pulumi = "^3.48.0"
pulumi-gcp = "^6.44.0"
trc-hpc-lib-gke-cluster = {path = "../lib-gke-cluster", develop = true}
m

millions-pharmacist-626

12/22/2022, 7:27 PM
understood
out of curiosity, what's in
pyproject.toml
in the lib-gke-cluster ?
cause I see you also have a lockfile and a makefile
c

clever-painter-96148

12/22/2022, 7:29 PM
[tool.poetry]
name = "trc-hpc-lib-gke-cluster"
version = "0.0.0"
authors = ["x <x@x>"]
description = "GKE cluster deployment library"

[tool.poetry.dependencies]
python = "^3.10"
pulumi = "^3.48.0"
pulumi-gcp = "^6.44.0"
pulumi-kubernetes = "^3.23.1"

[tool.poetry.group.dev.dependencies]
black = "^22.12.0"
pyright = "^1.1.283"
pytest = "^7.2.0"
pylint = "^2.15.8"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.black]
line-length = 79

[tool.pyright]
typeCheckingMode = "strict"
I have a lockfile in every "package" of my monorepo, so in CI, I can "poetry install" every package individually
so I can run static analysis on them
but only when they change (vs. having 1 CI process that works on everything)
m

millions-pharmacist-626

12/22/2022, 7:30 PM
right makes sense
c

clever-painter-96148

12/22/2022, 7:31 PM
The Makefile is just to normalize CI commands:
$ cat lib-gke-cluster/Makefile
lint:
        pylint trc_hpc_lib_gke_cluster tests

test:
        pytest --verbose
$ cat project-gke-cluster/Makefile
lint:
        pylint *.py

test:
        true
because pylint is not happy looking at the whole pulumi program folder (it expects a python package) and because I do no write tests for the program and pytest fails when there are no tests
m

millions-pharmacist-626

12/22/2022, 7:32 PM
understood, tyvm. If you don't mind me asking, I have one last question: currently my requirements are much less complicated than that, as in i just want to share a common
consts.py
file among different projects and I didn't want to use Stack References. Do you think a similar approach makes sense?
c

clever-painter-96148

12/22/2022, 7:33 PM
If you are sure you'll grow in complexity very quickly, yes.
But your question reminds me a former project where we somewhat overengineered a few stuff, like we created a "company-constants" package. After 4 years, only a handful of useful stuff landed in there, but most of it was junk only used in a few places šŸ˜›
lesson learnt: avoid that kind of packages as long as possible...
that said, this specific case was painful because it was in a multirepo environment; in a monorepo environment, i would be less afraid as refactoring is always easier.
m

millions-pharmacist-626

12/22/2022, 7:36 PM
i don't think it's gonna grow that much more, at best it's gonna get an extra project but in itself it's relatively small. For info, it's a repo to deploy a bastion server on AWS EC2, the 2 pulumi projects are: ā€¢ the infra itself ā€¢ the IAM policies to manage the infra The 3rd project would have the codepipeline IaC. Basically what I did is enable the management of this project through CI/CD so that I can use a single pulumi account and everything is done through git commits and pull requests, the CI triggers on merge to main
so basically if you were to git clone the repo today and get started with it, you'd do: ā€¢
cd pipeline
and
pulumi up
ā€¢
cd iam
and
pulumi up
ā€¢ make any changes to
infra
and
git commit
+
git push
im thinking of sharing it soon here when it's fully polished so if u care I can send u the link once it's public
c

clever-painter-96148

12/22/2022, 7:40 PM
I believe that well documented Pulumi-based OSS projects are always welcomed and useful. I found help thanks to such repositories many many times. šŸ˜„
m

millions-pharmacist-626

12/22/2022, 7:40 PM
ah yeah i also need to write docs ehehe
alright thx for the help ! šŸ™‚
c

clever-painter-96148

12/22/2022, 7:40 PM
Happy to help šŸ™‚