proud-art-4139910/03/2023, 9:11 AM
in the project file. Especially does it switch to the virtual env prior to running the Pulumi program? As it seems it does not. Here's the context. I'm using Ubuntu 22.04 which uses Python 3.10 by default. I also installed Python 3.11 (from PPA) because that's the target runtime I use on AWS. I'm provisioning couple of Lambda functions and in the Pulumi program, I use
from Pulumi Command provider to prepare the distribution package for the Lambdas. The command installs the dependencies using
into a directory containing the handler and then uses the directory in
supplied to the Lambda resource. The problem is as follows. The default system Python is 3.10 and I can't simply change it because it breaks some things (e.g. APT package manager). I thus manually create the
directory for Pulumi with
. I expect that when running
python3.11 -m venv venv
, the Pulumi program runs in context of the virtual env, i.e. using
from the virtual env and also pointing the
binary to the one from virtual env. But it's not the case, it uses Python 3.10 system default instead. That's a problem, because one of the dependencies is Pydantic (via AWS Lambda Powertools) and it needs the installed version to exactly match the target runtime platform. It fails running the Lambda function, because the installed
package is for Python 3.10 while the runtime is Python 3.11. It fails with
If I temporarily switch the system default to Python 3.11 (using alternatives), it works and correctly installs the package for Python 3.11. I know I can manually switch to the virtual env prior to running
Unable to import module 'lambda_function': No module named 'pydantic_core._pydantic_core'
, it's just about the expectations. Thanks
dry-keyboard-9479510/03/2023, 9:30 AM
proud-art-4139910/03/2023, 9:31 AM
in the path.
dry-keyboard-9479510/03/2023, 9:34 AM
gives you Python 3.11.x?
proud-art-4139910/03/2023, 9:34 AM
dry-keyboard-9479510/03/2023, 9:37 AM
, it's installing the wrong versions, or that pulumi isn't using the venv?
proud-art-4139910/03/2023, 9:40 AM
is a Python script which contains a shebang with path to the Python interpreter. When Pulumi is not using the
uses the wrong Python interpreter (on a system path)
dry-keyboard-9479510/03/2023, 9:48 AM
(ie, use the venv in current shell), it all works. But without, pulumi defaults to using the system python env for running and installing. Can you post the contents of
, can redact anything sensitive?
side, I wasn't aware pulumi did dependency management. Always managed my own venvs 🙂
proud-art-4139910/03/2023, 9:51 AM
so if you doYes, exactly.(ie, use the venv in current shell), it all works.
But without, pulumi defaults to using the system python env for running and installing.That's not exactly what I stated (and hence the question). Pulumi clearly uses the
for installing the dependencies and also interpreting the Pulumi program. It just doesn't switch to the virtualenv. (And thus, when using
with command like
, it uses the
pip install ...
on system path instead the one in the virtualenv.)
name: xxx-prod description: xxx runtime: name: python options: virtualenv: venv options: refresh: always
dry-keyboard-9479510/03/2023, 10:01 AM
proud-art-4139910/03/2023, 10:07 AM
would most probably run the command in a subprocess... Thanks!
dry-keyboard-9479510/03/2023, 1:15 PM
should use the venv as well. It's worth raising a feature request. I've similar problems with passing Google Auth tokens around, and it'd be good to see functionality to remove this kind of boilerplate
great-sunset-35511/04/2023, 11:20 PM
config at all because I've had multiple projects using the same VENV and just used
before I started working. Nevertheless, I also made a mistake of using pedantic for
which resulted in a lot of unnecessary type headaches with outputs. Apart from that I used ASDF to manage my python installations and sometimes Pulumi just did not register the PATH to python binary correctly 🤷 (perhaps it has to do something with multiple PyCharm windows I had opened) In the end, I decided to move to TypeScript because it is much easier to express and write for pulumi. Mainly due to required boilerplate of for python classes (this ma be a gamechanger though https://github.com/pulumi/pulumi/issues/11732) As for lambdas, I find your solution interesting. I settled on dissociating the artifact from pulumi and just give pulumi an S3 path of pre-built package, so that lambda can use any runtime and I can write a more generic/flexible component.