diff --git a/docs/development/ADRs/eve/0001-Runtime_Resolution_of_PEP695_Type_Aliases.md b/docs/development/ADRs/eve/0001-Runtime_Resolution_of_PEP695_Type_Aliases.md new file mode 100644 index 0000000000..1b8707a18b --- /dev/null +++ b/docs/development/ADRs/eve/0001-Runtime_Resolution_of_PEP695_Type_Aliases.md @@ -0,0 +1,70 @@ +--- +tags: [typing, datamodels] +--- + +# Runtime Resolution of PEP 695 Type Aliases + +- **Status**: valid +- **Authors**: Enrique González Paredes (@egparedes) +- **Created**: 2026-08-05 +- **Updated**: 2026-08-05 + +In the context of users writing `type MyField = Field[...]` in datamodel and DSL annotations, facing the fact that `TypeAliasType` is an opaque object which no runtime introspection helper unwraps, we decided to resolve aliases at the existing annotation-dispatch funnels through a single `eve.extended_typing.eval_type_alias` helper, to achieve uniform support at every nesting depth without eagerly rewriting stored annotations, accepting that an alias whose value is not yet defined is only validated on first instantiation. + +## Context + +Python 3.12 (PEP 695) introduces `type X = ...`, which creates a `typing.TypeAliasType` object instead of binding the annotation directly. The GT4Py Python floor is now 3.12, so users can and will write these aliases. + +Three properties make this awkward at runtime: + +- `typing.get_type_hints` (and therefore `eve.extended_typing.get_partial_type_hints`) returns the `TypeAliasType` object unresolved. It never evaluates `__value__`, so the existing `NameError`-to-`ForwardRef` fallback never triggers for aliases. +- `__value__` is evaluated lazily and raises `NameError` until every name it mentions exists. An alias may legitimately be defined before its target. +- `gt4py.eve.extended_typing` re-exports both `typing` and `typing_extensions` with star imports, so the name `TypeAliasType` resolves to `typing_extensions.TypeAliasType`. That class is _not_ the class of a native `type X = ...` alias, so a plain `isinstance(x, xtyping.TypeAliasType)` check silently fails to match. Additionally, `MyGenericAlias[int]` is a `types.GenericAlias` which proxies attribute lookups to its origin, so `hasattr(x, "__value__")` is true for it as well while `isinstance()` is not. + +Drivers: + +- Nested annotations (`list[MyAlias]`, `tuple[MyAlias, ...]`, `Optional[MyAlias]`) have to work, not only top-level ones. +- Neither `gt4py.eve` nor `gt4py.next` should grow scattered `isinstance` checks against a moving target. +- The failure mode for a genuinely broken annotation has to stay a normal, understandable error. + +Note that PEP 695 _generics_ (`class Model[T](DataModel)`) already work unchanged, because `__parameters__` is still populated for them. Only the alias form is affected. + +## Decision + +1. `gt4py.eve.extended_typing` owns the concept, exposing `is_type_alias(obj)` (checking both the `typing` class and the `typing_extensions` backport) and `eval_type_alias(annotation)`. The latter follows alias chains, substitutes type parameters for parametrized generic aliases, returns the identical object for non-aliases, and raises `TypeError` for recursive or over-nested aliases. +2. Resolution happens at the **annotation-dispatch funnels**, not at annotation-storage time: `eve.type_validation.SimpleTypeValidatorFactory.__call__` and `eve.datamodels.core._make_type_converter`. Both already recurse into type arguments, so nesting is covered for free, and the stored annotation keeps the alias name for reprs and introspection. +3. A `NameError` from `eval_type_alias` is a **signal, not an error** inside `eve`: `type_validation` lets it propagate, and `datamodels.field_type_validator_factory` catches it and installs the existing `ForwardRefValidator`, deferring validation to the first instantiation, which is the same treatment string forward references already get. + +## Consequences + +Easier: + +- Any annotation reachable through the dispatch funnels supports aliases at any nesting depth with no further work. +- Only one place has to change if CPython or `typing_extensions` alters the alias implementation. + +Harder, and accepted: + +- A field annotated with an alias whose value is not yet resolvable is validated on first instantiation instead of at class definition, so its errors surface later than for other fields. Its error message also reports the bare field name rather than the qualified `Model.field` one, matching the pre-existing behavior for forward references. +- `ClassVar` hidden behind an alias (`type CV = ClassVar[int]`) is not detected as a class variable by `datamodels`. This is not supported and no attempt is made to detect it. +- The resolution depth is capped (64 steps) so that recursive aliases fail fast instead of hanging. + +## Alternatives considered + +### Rewrite resolved annotations eagerly in `_make_datamodel` + +Unwrap aliases right after `get_partial_type_hints` and store the resolved annotation on the field. + +- Good, because every downstream consumer sees a plain annotation with no further changes. +- Bad, because it forces eager evaluation of `__value__`, breaking aliases which reference names defined later. +- Bad, because it destroys the alias name in `__annotations__`, field reprs and generated `__init__` signatures. + +### `isinstance` checks at each call site + +- Good, because it needs no new API. +- Bad, because the correct check is non-obvious (two distinct classes, and `hasattr` is a trap) and would be re-derived wrongly at each site. +- Bad, because it goes against the ruff `typing-modules = ['gt4py.eve.extended_typing']` convention that typing concepts live in a single module. + +## References + +- [PEP 695 - Type Parameter Syntax](https://peps.python.org/pep-0695/) +- `src/gt4py/eve/extended_typing.py` (`is_type_alias`, `eval_type_alias`) diff --git a/docs/development/ADRs/eve/README.md b/docs/development/ADRs/eve/README.md new file mode 100644 index 0000000000..8bc4f5ffc4 --- /dev/null +++ b/docs/development/ADRs/eve/README.md @@ -0,0 +1,19 @@ +# Architecture Decision Records Index (`gt4py.eve`) + +This document contains links to all _Architecture Decision Record_ (ADR) documents written for the `gt4py.eve` framework. The [top-level README](../README.md) explains when and why we write ADRs. + +## How to write ADRs + +See [top-level README](../README.md) on when and why we write ADRs. + +Writing a new ADR is simple: + +1. Use the existing [Template](../Template.md) as an ice-breaker to start a new ADR file, but modify it and simplify it as much as possible to fit the type of decision being documented. If extra files (e.g. images) are needed for whatever reason, add them to the `_static/` folder. +2. Add a link to the new ADR file to the fitting topic in the index section below. +3. Open a PR to merge the changes into the main branch and let the team know about the new ADR. + +## Index by Topic + +### Type system and annotations #typing + +- [0001 - Runtime Resolution of PEP 695 Type Aliases](0001-Runtime_Resolution_of_PEP695_Type_Aliases.md) diff --git a/src/gt4py/eve/datamodels/core.py b/src/gt4py/eve/datamodels/core.py index 30232ec2c6..df460077dc 100644 --- a/src/gt4py/eve/datamodels/core.py +++ b/src/gt4py/eve/datamodels/core.py @@ -224,11 +224,18 @@ def _field_type_validator_factory(type_annotation: TypeAnnotation, name: str) -> """Field type validator for datamodels, supporting forward references.""" if isinstance(type_annotation, ForwardRef): return ForwardRefValidator(factory) - else: + + try: simple_validator = factory(type_annotation, name, required=True) - return ValidatorAdapter( - simple_validator, f"{getattr(simple_validator, '__name__', 'TypeValidator')}" - ) + except NameError: + # A PEP 695 type alias ('type X = ...') is evaluated lazily and may + # reference a name which does not exist yet at class creation time. + # Defer the creation of the validator, as done for forward references. + return ForwardRefValidator(factory) + + return ValidatorAdapter( + simple_validator, f"{getattr(simple_validator, '__name__', 'TypeValidator')}" + ) return _field_type_validator_factory @@ -972,6 +979,9 @@ def __class_getitem__( def _make_type_converter(type_annotation: TypeAnnotation, name: str) -> TypeConverter[_T]: # TODO(egparedes): if a "typing tree" structure is implemented, refactor this code # as a tree traversal. + if (resolved_annotation := xtyping.eval_type_alias(type_annotation)) is not type_annotation: + return _make_type_converter(resolved_annotation, name) + if xtyping.is_actual_type(type_annotation) and not isinstance(None, type_annotation): assert not xtyping.get_args(type_annotation) assert isinstance(type_annotation, type) diff --git a/src/gt4py/eve/extended_typing.py b/src/gt4py/eve/extended_typing.py index 6470befe1a..e7f2df94e4 100644 --- a/src/gt4py/eve/extended_typing.py +++ b/src/gt4py/eve/extended_typing.py @@ -162,6 +162,7 @@ def is_maybe_nested_in_tuple_of( Type, _types.GenericAlias, _typing._BaseGenericAlias, # type: ignore[name-defined] # _BaseGenericAlias is not exported in stub + _typing.TypeAliasType, ] SolvedTypeAnnotation = Union[SingleTypeAnnotation, _typing._SpecialForm] @@ -408,6 +409,69 @@ def is_actual_type(obj: Any) -> TypeGuard[Type]: ) +#: Both implementations of PEP 695 type aliases. They are distinct classes, and a native +#: ``type X = ...`` alias is not an instance of the ``typing_extensions`` backport (nor +#: the other way round), so both of them have to be checked. +_TypeAliasTypes: Final[tuple[type, ...]] = ( + _typing.TypeAliasType, + _typing_extensions.TypeAliasType, +) + +#: Upper bound for the number of resolution steps in `eval_type_alias`, to avoid +#: hanging on recursive aliases like ``type A = A``. +_MAX_TYPE_ALIAS_DEPTH: Final = 64 + + +def is_type_alias(obj: Any) -> TypeGuard[TypeAliasType]: + """Check if an object is a PEP 695 type alias (``type X = ...``).""" + return isinstance(obj, _TypeAliasTypes) + + +def eval_type_alias(annotation: Any) -> Any: + """Replace a PEP 695 type alias by the annotation it stands for. + + Chained aliases are followed until a non-alias annotation is reached, and + parametrized generic aliases (``MyAlias[int]``) get their type parameters + substituted. Any other annotation is returned unchanged (as the identical + object), so callers can use an identity check to find out whether anything + was actually resolved. + + Note that alias values are evaluated lazily by the interpreter, so this is + the point where the names used in the alias definition are looked up for + the first time. + + Args: + annotation: Any type annotation. + + Returns: + The annotation the alias stands for, or `annotation` itself. + + Raises: + NameError: If the alias value references a name which is not defined yet. + TypeError: If the alias is recursive, nested too deeply, or cannot be + parametrized with the given type arguments. + + Examples: + >>> type MyInt = int + >>> eval_type_alias(MyInt) + + + >>> eval_type_alias(float) + + """ + for _ in range(_MAX_TYPE_ALIAS_DEPTH): + if is_type_alias(annotation): + annotation = annotation.__value__ + elif is_type_alias(alias := get_origin(annotation)): + annotation = alias.__value__[get_args(annotation)] + else: + return annotation + + raise TypeError( + f"Type alias '{annotation}' cannot be resolved (recursive or nested too deeply)." + ) + + if hasattr(_typing_extensions, "Any") and _typing.Any is not _typing_extensions.Any: # type: ignore[attr-defined] # _typing_extensions.Any only from >= 4.4 # When using Python < 3.11 and typing_extensions >= 4.4 there are # two different implementations of `Any` diff --git a/src/gt4py/eve/type_validation.py b/src/gt4py/eve/type_validation.py index ec66be2307..1c4a3ae673 100644 --- a/src/gt4py/eve/type_validation.py +++ b/src/gt4py/eve/type_validation.py @@ -199,6 +199,20 @@ def __call__( ): # see https://github.com/python/cpython/issues/105499 type_annotation = typing.Union[type_annotation.__args__] + # PEP 695 type aliases ('type X = ...') stand for another annotation whose + # value is only evaluated on demand. A 'NameError' raised here means the + # alias references a name which does not exist yet, and is deliberately not + # converted: 'datamodels' defers the creation of the validator when it sees + # one, in the same way it does for forward references. + try: + resolved_annotation = xtyping.eval_type_alias(type_annotation) + except TypeError as error: + raise exceptions.EveValueError( + f"{type_annotation} type annotation is not supported." + ) from error + if resolved_annotation is not type_annotation: + return make_recursive(resolved_annotation) + # Non-generic types if xtyping.is_actual_type(type_annotation): assert not xtyping.get_args(type_annotation) diff --git a/tests/eve_tests/unit_tests/test_datamodels.py b/tests/eve_tests/unit_tests/test_datamodels.py index d2a1564fc2..af50be09e5 100644 --- a/tests/eve_tests/unit_tests/test_datamodels.py +++ b/tests/eve_tests/unit_tests/test_datamodels.py @@ -1287,3 +1287,70 @@ class Plain: assert isinstance(Model(value=1), datamodels.DataModelABC) assert not issubclass(Plain, datamodels.DataModelABC) assert not isinstance(Plain(), datamodels.DataModelABC) + + +# -- PEP 695 type aliases -- +type SampleAliasInt = int +type SampleAliasNested = typing.List[SampleAliasInt] +type SampleAliasPair[T] = typing.Tuple[T, T] + + +def test_type_alias_field(): + class Model(datamodels.DataModel): + value: SampleAliasInt + + assert Model(value=1).value == 1 + with pytest.raises(TypeError, match="Model.value"): + Model(value="1") + + +def test_nested_type_alias_field(): + class Model(datamodels.DataModel): + direct: SampleAliasNested + inline: typing.List[SampleAliasInt] + parametrized: SampleAliasPair[int] + + model = Model(direct=[1], inline=[2], parametrized=(3, 4)) + assert model.direct == [1] + + with pytest.raises(TypeError, match="Model.inline"): + Model(direct=[1], inline=["2"], parametrized=(3, 4)) + + +def test_coerced_type_alias_field(): + class Model(datamodels.DataModel): + value: datamodels.Coerced[SampleAliasInt] + + assert Model(value="42").value == 42 + + +def test_type_alias_field_with_undefined_value_is_deferred(): + # The alias value is evaluated lazily, so the class body must not fail even + # though 'DefinedLater' does not exist yet. Validation happens on the first + # instantiation, like for forward references. + # + # This is 'exec'-ed instead of written inline because this test module uses + # 'from __future__ import annotations', which would turn the annotation into + # the string 'LazyAlias' and exercise the plain forward-reference path + # instead of the type-alias one. 'dont_inherit=True' is required for the + # same reason: 'exec()' inherits the '__future__' flags of the calling module. + source = """ +from gt4py.eve import datamodels + +type LazyAlias = DefinedLater + +class Model(datamodels.DataModel): + value: LazyAlias + +class DefinedLater: + pass +""" + namespace: Dict[str, Any] = {} + exec(compile(source, "", "exec", dont_inherit=True), namespace) + Model, DefinedLater = namespace["Model"], namespace["DefinedLater"] + + assert isinstance(Model(value=DefinedLater()).value, DefinedLater) + # Note: the deferred validator reports the bare field name instead of the + # qualified 'Model.value', as it does for regular forward references. + with pytest.raises(TypeError, match="'value' must be"): + Model(value=1) diff --git a/tests/eve_tests/unit_tests/test_extended_typing.py b/tests/eve_tests/unit_tests/test_extended_typing.py index b6e4896866..42e5a28ba9 100644 --- a/tests/eve_tests/unit_tests/test_extended_typing.py +++ b/tests/eve_tests/unit_tests/test_extended_typing.py @@ -14,6 +14,7 @@ import typing import pytest +import typing_extensions from gt4py.eve import extended_typing as xtyping from gt4py.eve.extended_typing import ( @@ -416,3 +417,50 @@ def test_is_single_dispatch_callable(): # Plain callables and non-callables are rejected. assert not xtyping.is_single_dispatch_callable(lambda _: None) assert not xtyping.is_single_dispatch_callable(42) + + +# -- PEP 695 type aliases -- +type SampleIntAlias = int +type SampleChainedAlias = SampleIntAlias +type SampleGenericAlias[T] = Tuple[T, T] +type SampleRecursiveAlias = SampleRecursiveAlias + + +def test_is_type_alias(): + # Both the native 'typing' class and the 'typing_extensions' backport have to be + # recognized: they are distinct classes and a native alias is not an instance + # of the backport. + assert xtyping.is_type_alias(SampleIntAlias) + assert xtyping.is_type_alias(typing_extensions.TypeAliasType("Backported", str)) + + assert not xtyping.is_type_alias(int) + assert not xtyping.is_type_alias(List[int]) + assert not xtyping.is_type_alias(SampleGenericAlias[int]) + + +def test_eval_type_alias(): + assert xtyping.eval_type_alias(SampleIntAlias) is int + assert xtyping.eval_type_alias(SampleChainedAlias) is int + assert xtyping.eval_type_alias(SampleGenericAlias[int]) == Tuple[int, int] + + +def test_eval_type_alias_passes_through_non_aliases(): + for annotation in (int, List[int], xtyping.Any, None): + assert xtyping.eval_type_alias(annotation) is annotation + + +def test_eval_type_alias_with_undefined_value(): + # Alias values are evaluated lazily, so the name is only looked up here. + type LazyAlias = _defined_later # noqa: F821 [undefined-name] # defined below + + with pytest.raises(NameError): + xtyping.eval_type_alias(LazyAlias) + + _defined_later = int + + assert xtyping.eval_type_alias(LazyAlias) is int + + +def test_eval_type_alias_with_recursive_alias(): + with pytest.raises(TypeError, match="cannot be resolved"): + xtyping.eval_type_alias(SampleRecursiveAlias) diff --git a/tests/eve_tests/unit_tests/test_type_validation.py b/tests/eve_tests/unit_tests/test_type_validation.py index 6b119a5252..faccacdfcc 100644 --- a/tests/eve_tests/unit_tests/test_type_validation.py +++ b/tests/eve_tests/unit_tests/test_type_validation.py @@ -157,6 +157,25 @@ class SampleSlottedDataClass: ) +# -- PEP 695 type aliases -- +type SampleIntAlias = int +type SampleChainedAlias = SampleIntAlias +type SampleListAlias = List[SampleIntAlias] +type SamplePairAlias[T] = Tuple[T, T] + + +SAMPLE_TYPE_DEFINITIONS.extend( + [ + (SampleIntAlias, [1, -1], [1.0, "1"], None, None), + (SampleChainedAlias, [1, -1], [1.0, "1"], None, None), + (SampleListAlias, ([1, 2, 3], []), (1, [1.0]), None, None), + (List[SampleIntAlias], ([1, 2, 3], []), (1, [1.0]), None, None), + (Optional[SampleIntAlias], [1, None], ["1"], None, None), + (SamplePairAlias[int], [(1, 2)], [(1, "2"), (1,)], None, None), + ] +) + + @pytest.mark.parametrize("validator", VALIDATORS) @pytest.mark.parametrize( ["type_hint", "valid_values", "wrong_values", "globalns", "localns"], SAMPLE_TYPE_DEFINITIONS @@ -253,3 +272,29 @@ def test_simple_validation_particularities(): with pytest.raises(ValueError, match="annotation is not supported"): type_val.simple_type_validator_factory(InvalidAnnotation, "value") + + +def test_recursive_type_alias_is_not_supported(): + type RecursiveAlias = RecursiveAlias + + with pytest.raises(ValueError, match="annotation is not supported"): + type_val.simple_type_validator_factory(RecursiveAlias, "value") + + assert type_val.simple_type_validator_factory(RecursiveAlias, "value", required=False) is None + + +def test_type_alias_with_undefined_value_propagates_name_error(): + # Alias values are evaluated lazily, so the missing name only shows up here. + # The error must not be swallowed: 'datamodels' relies on it to defer the + # creation of the validator. + type LazyAlias = _defined_later # noqa: F821 [undefined-name] # defined below + + with pytest.raises(NameError): + type_val.simple_type_validator_factory(LazyAlias, "value") + + _defined_later = int + + validator = type_val.simple_type_validator_factory(LazyAlias, "value") + validator(1) + with pytest.raises(TypeError): + validator("1")