https://pulumi.com logo
#python
Title
# python
e

echoing-postman-88590

11/27/2023, 4:30 PM
Hi, I would like to use pydantic with pulumi. Consider the following example:
Copy code
from pydantic.dataclasses import dataclass
import pulumi_gcp as gcp

@dataclass
class AArgs:
    service_account: gcp.serviceaccount.Account
on
pulumi preview
I get:
Copy code
@dataclass
         ^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/dataclasses.py", line 235, in dataclass
        return create_dataclass(_cls)
               ^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/dataclasses.py", line 226, in create_dataclass
        pydantic_complete = _pydantic_dataclasses.complete_dataclass(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_dataclasses.py", line 153, in complete_dataclass
        schema = gen_schema.generate_schema(cls, from_dunder_get_core_schema=False)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 468, in generate_schema
        schema = self._generate_schema(obj)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 700, in _generate_schema
        schema = self._post_process_generated_schema(self._generate_schema_inner(obj))
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 727, in _generate_schema_inner
        return self.match_type(obj)
               ^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 801, in match_type
        return self._dataclass_schema(obj, None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1398, in _dataclass_schema
        args = sorted(
               ^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1399, in <genexpr>
        (self._generate_dc_field_schema(k, v, decorators) for k, v in fields.items()),
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 903, in _generate_dc_field_schema
        common_field = self._common_field_schema(name, field_info, decorators)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 951, in _common_field_schema
        schema = self._apply_annotations(
                 ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1654, in _apply_annotations
        schema = get_inner_schema(source_type)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in __call__
        schema = self._handler(__source_type)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1635, in inner_handler
        schema = self._generate_schema(obj)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 700, in _generate_schema
        schema = self._post_process_generated_schema(self._generate_schema_inner(obj))
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 727, in _generate_schema_inner
        return self.match_type(obj)
               ^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 814, in match_type
        return self._unknown_type_schema(obj)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 366, in _unknown_type_schema
        raise PydanticSchemaGenerationError(
    pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class 'pulumi_gcp.serviceaccount.account.Account'>. Set `arbitrary_types_allowed=True` in the model_config to ignore this error or implement `__get_pydantic_core_schema__` on your type to fully support it.
    
    If you got this error by calling handler(<some type>) within `__get_pydantic_core_schema__` then you likely need to call `handler.generate_schema(<some type>)` since we do not call `__get_pydantic_core_schema__` on `<some type>` otherwise to avoid infinite recursion.
    
    For further information visit <https://errors.pydantic.dev/2.5/u/schema-for-unknown-type>
Do I need to install anything else? Thanks
b

billowy-army-68599

11/27/2023, 4:37 PM
I think it’s telling you how to fix this:
Set
arbitrary_types_allowed=True
in the model_config to ignore this error or implement
__get_pydantic_core_schema__
on your type to fully support it.
this is a good example of using python with pulumi and pydantic: https://github.com/rolindroy/ol-infrastructure/blob/6bf55e2f0d3f44d42fbea476250c9b7c632a7860/src/ol_infrastructure/components/aws/database.py They set
arbitrary_types_allowed = True
d

dry-keyboard-94795

11/27/2023, 4:39 PM
See instructions in the pydantic docs for how to set the config on a dataclass: https://docs.pydantic.dev/latest/concepts/dataclasses/#dataclass-config
e

echoing-postman-88590

11/27/2023, 4:42 PM
Thanks for the reply, but is there a way to avoid to set that. It is a bit against the aim of pydantic
b

billowy-army-68599

11/27/2023, 4:43 PM
See Anthony’s reply
e

echoing-postman-88590

11/27/2023, 4:46 PM
Yes, I understand that I need to set ``arbitrary_types_allowed = True``. May question is more about if I allow arbitrary types it is a bit against the aim of pydantic?
d

dry-keyboard-94795

11/27/2023, 4:49 PM
Do you mean to pass arbitrary dictionaries into the dataclass, and have it turn the input into the Account object?
e

echoing-postman-88590

11/27/2023, 4:51 PM
Yes, I would expect pydantic to be able to check you passed an
gcp.serviceaccount.Account
for this item, etc
d

dry-keyboard-94795

11/27/2023, 4:54 PM
Arbitrary types still does an instance check on the input, so will work as expected. It's a flag to allow bypassing certain parts of the schema generation, as pydantic won't know the expected serialised input for json support
e

echoing-postman-88590

11/27/2023, 4:54 PM
ok thanks for the explanation
d

dry-keyboard-94795

11/27/2023, 4:56 PM
If you're looking to logically group resources together, pulumi has support for it using Component Resources: https://www.pulumi.com/docs/concepts/resources/components/
e

echoing-postman-88590

11/27/2023, 4:58 PM
Yes, I am using the
ComponentResource
class, I am using pydantic to expand and check and expand from json the
args
part
g

great-sunset-355

12/05/2023, 8:05 AM
@echoing-postman-88590 I'd strongly advise against using pulumi with Pydantic because you will run into many problems with
outputs
and other type annotations because Pydantic tries to "execute" the type to check it.
arbirary_types_allowed
may work at the beginning but trust me it goes bad relatively fast. I started with the style of the repository that Jaxx has linked, however, it soon became clear that it has many limits. At the same time, I had another codebase with TypeScript which has its own quirks but in general it is more pleasant for larger projects. (IMO) I believe that Pulumi Python will become good once typeddicts are implemented https://github.com/pulumi/pulumi/issues/11732
a

adventurous-butcher-54166

01/23/2024, 3:16 PM
I'm eager to find the best way forward for config mgmt / type hints... Found this Slack thread after coming across MIT's
ol_infrastructure
project and saw how they're using Pydantic. I went on a similar journey few months back in an effort to standardize config schemas across (Pulumi) projects, resource components etc. and support type hints. Ended up using dataclasses and created a configuration loader which automatically inspects type annotations and populates the config object from a Pulumi stack config using relevant
pulumi.Config
getter functions, with support for: • Supports various types... strings, int, bool, dict, list, tuple • Optional (
Config.get_x
) and non-optional (
Config.require_x
) • Secrets for all types (
Config.get_x_secret
/
Config.require_x_secret
) • Populating with default values defined int he dataclass schema when a config key is not present in a stack Sharing some snippets here should anyone be interested in having a look... https://gist.github.com/olafurnielsen/60f3d8534dfea5007b6ccbf290457f43 I still feel like I haven't cracked this. Was inspired by the way ol_infrastructure used Pydantic for input validation etc. but curious to know the limitations you hit @great-sunset-355 Would also love to discuss this with some clever minds any time 🙂
d

dry-keyboard-94795

01/23/2024, 3:23 PM
Your gist link doesn't work, seems to have an extra character on the end
a

adventurous-butcher-54166

01/23/2024, 3:24 PM
@dry-keyboard-94795 - sorry fixed 🙂
To summarize, I'm unsure whether to: • Continue on the dataclass path • Migrate to Pydantic schemas • Or wait for how Pulumi will go forward with Improve typing of resource arguments in Python
d

dry-keyboard-94795

01/23/2024, 6:33 PM
The linked issue is around improving resources in python, they won't be touching on configs there
My usual approach with typing configs is to just have a config module, and manually type any untyped configs and stackreferences there. I don't mind a bit of messiness so long as it's confined, and the rest of the code gets hints
However I can see the benefit to having something like a dataclass (or structs in golang/typescript) to actually use code to define the schema for stack configs. My thinking in python's terms would be to have a pydantic class that during a stack refresh would update the schema definition in Pulumi.yaml based on initialisation of your Config class. It'd be expected that each Provider has one of these Config classes too. Im unsure though, just thinking through usecases for having an actual config class
a

adventurous-butcher-54166

01/24/2024, 10:47 AM
Yeah, we're using the
pulumi_azure_native
provider and most if not all resource components take in keyword arguments but can alse be passed an Args class. I was thinking to align the config schema with those arguments to easily pass them in using same keyword names and structure to reduce the amount of "parameter passing around" in our IaC library... Also making some experiments with Pulumi ESC, especially excited about the ability to extract stack outputs and have them as direct configuration inputs in another stack.
Ended up going full force down the Pydantic road. Wrote a Pydantic Settings extension for auto-populating non-initialized values of the datamodel via pulumi.Config. Knows how to deal with pulumi Outputs and fetch (even nested) stack reference outputs. As a bonus it will also generate a local JSON schema and annotates Pulumi stack yamls with a oneliner referencing that schema for real time validation feedback in the IDE.
3 Views