Hi, I would like to use pydantic with pulumi. Consider the following example:

from pydantic.dataclasses import dataclass
import pulumi_gcp as gcp

@dataclass
class AArgs:
    service_account: gcp.serviceaccount.Account

on

pulumi preview

I get:

@dataclass
         ^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/dataclasses.py", line 235, in dataclass
        return create_dataclass(_cls)
               ^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/dataclasses.py", line 226, in create_dataclass
        pydantic_complete = _pydantic_dataclasses.complete_dataclass(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_dataclasses.py", line 153, in complete_dataclass
        schema = gen_schema.generate_schema(cls, from_dunder_get_core_schema=False)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 468, in generate_schema
        schema = self._generate_schema(obj)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 700, in _generate_schema
        schema = self._post_process_generated_schema(self._generate_schema_inner(obj))
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 727, in _generate_schema_inner
        return self.match_type(obj)
               ^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 801, in match_type
        return self._dataclass_schema(obj, None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1398, in _dataclass_schema
        args = sorted(
               ^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1399, in <genexpr>
        (self._generate_dc_field_schema(k, v, decorators) for k, v in fields.items()),
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 903, in _generate_dc_field_schema
        common_field = self._common_field_schema(name, field_info, decorators)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 951, in _common_field_schema
        schema = self._apply_annotations(
                 ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1654, in _apply_annotations
        schema = get_inner_schema(source_type)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in __call__
        schema = self._handler(__source_type)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1635, in inner_handler
        schema = self._generate_schema(obj)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 700, in _generate_schema
        schema = self._post_process_generated_schema(self._generate_schema_inner(obj))
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 727, in _generate_schema_inner
        return self.match_type(obj)
               ^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 814, in match_type
        return self._unknown_type_schema(obj)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 366, in _unknown_type_schema
        raise PydanticSchemaGenerationError(
    pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class 'pulumi_gcp.serviceaccount.account.Account'>. Set `arbitrary_types_allowed=True` in the model_config to ignore this error or implement `__get_pydantic_core_schema__` on your type to fully support it.
    
    If you got this error by calling handler(<some type>) within `__get_pydantic_core_schema__` then you likely need to call `handler.generate_schema(<some type>)` since we do not call `__get_pydantic_core_schema__` on `<some type>` otherwise to avoid infinite recursion.
    
    For further information visit <https://errors.pydantic.dev/2.5/u/schema-for-unknown-type>

Do I need to install anything else? Thanks

I think it’s telling you how to fix this:

Set

arbitrary_types_allowed=True

in the model_config to ignore this error or implement

__get_pydantic_core_schema__

on your type to fully support it.

See instructions in the pydantic docs for how to set the config on a dataclass: https://docs.pydantic.dev/latest/concepts/dataclasses/#dataclass-config

Thanks for the reply, but is there a way to avoid to set that. It is a bit against the aim of pydantic

See Anthony’s reply

Yes, I understand that I need to set ``arbitrary_types_allowed = True``. May question is more about if I allow arbitrary types it is a bit against the aim of pydantic?

Do you mean to pass arbitrary dictionaries into the dataclass, and have it turn the input into the Account object?

Yes, I would expect pydantic to be able to check you passed an

gcp.serviceaccount.Account

for this item, etc

Arbitrary types still does an instance check on the input, so will work as expected. It's a flag to allow bypassing certain parts of the schema generation, as pydantic won't know the expected serialised input for json support

ok thanks for the explanation

If you're looking to logically group resources together, pulumi has support for it using Component Resources: https://www.pulumi.com/docs/concepts/resources/components/

Yes, I am using the

ComponentResource

class, I am using pydantic to expand and check and expand from json the

args

part

@echoing-postman-88590 I'd strongly advise against using pulumi with Pydantic because you will run into many problems with

outputs

and other type annotations because Pydantic tries to "execute" the type to check it.

arbirary_types_allowed

may work at the beginning but trust me it goes bad relatively fast. I started with the style of the repository that Jaxx has linked, however, it soon became clear that it has many limits. At the same time, I had another codebase with TypeScript which has its own quirks but in general it is more pleasant for larger projects. (IMO) I believe that Pulumi Python will become good once typeddicts are implemented https://github.com/pulumi/pulumi/issues/11732

I'm eager to find the best way forward for config mgmt / type hints... Found this Slack thread after coming across MIT's

ol_infrastructure

project and saw how they're using Pydantic. I went on a similar journey few months back in an effort to standardize config schemas across (Pulumi) projects, resource components etc. and support type hints. Ended up using dataclasses and created a configuration loader which automatically inspects type annotations and populates the config object from a Pulumi stack config using relevant

pulumi.Config

getter functions, with support for: • Supports various types... strings, int, bool, dict, list, tuple • Optional (

Config.get_x

) and non-optional (

Config.require_x

) • Secrets for all types (

Config.get_x_secret

/

Config.require_x_secret

) • Populating with default values defined int he dataclass schema when a config key is not present in a stack Sharing some snippets here should anyone be interested in having a look... https://gist.github.com/olafurnielsen/60f3d8534dfea5007b6ccbf290457f43 I still feel like I haven't cracked this. Was inspired by the way ol_infrastructure used Pydantic for input validation etc. but curious to know the limitations you hit @great-sunset-355 Would also love to discuss this with some clever minds any time 🙂

Your gist link doesn't work, seems to have an extra character on the end

@dry-keyboard-94795 - sorry fixed 🙂

The linked issue is around improving resources in python, they won't be touching on configs there

Yeah, we're using the

pulumi_azure_native

provider and most if not all resource components take in keyword arguments but can alse be passed an Args class. I was thinking to align the config schema with those arguments to easily pass them in using same keyword names and structure to reduce the amount of "parameter passing around" in our IaC library... Also making some experiments with Pulumi ESC, especially excited about the ability to extract stack outputs and have them as direct configuration inputs in another stack.