PEP 718: Specify binding, parametrisation and overload interactions

peps/pep-0718.rst

@@ -1,6 +1,6 @@
 PEP: 718
 Title: Subscriptable functions
-Author: James Hilton-Balfe <gobot1234yt@gmail.com>
+Author: James Hilton-Balfe <gobot1234yt@gmail.com>, Pablo Ruiz Cuevas <pablo.r.c@live.com>
 Sponsor: Guido van Rossum <guido@python.org>
 Discussions-To: https://discuss.python.org/t/28457/
 Status: Draft
@@ -17,41 +17,113 @@ This PEP proposes making function objects subscriptable for typing purposes. Doi
 gives developers explicit control over the types produced by the type checker where
 bi-directional inference (which allows for the types of parameters of anonymous
 functions to be inferred) and other methods than specialisation are insufficient. It
-also brings functions in line with regular classes in their ability to be
-subscriptable.
+also makes functions consistent with regular classes in their ability to be
+subscripted.
 
 Motivation
 ----------
 
-Unknown Types
-^^^^^^^^^^^^^
+Currently, classes allow passing type annotations for generic containers, this
+is especially useful in common constructors such as ``list``\, ``tuple`` and ``dict``
+etc.
 
-Currently, it is not possible to infer the type parameters to generic functions in
-certain situations:
+.. code-block:: python
+
+   my_integer_list = list[int]()
+   reveal_type(my_integer_list)  # type is list[int]
+
+At runtime ``list[int]`` returns a ``GenericAlias`` that can be later called, returning
+an empty list.
+
+Another example of this is creating a specialised ``dict`` type for a section of our
+code where we want to ensure that keys are ``str`` and values are ``int``:
+
+.. code-block:: python
+
+   NameNumberDict = dict[str, int]
+
+   NameNumberDict(
+      one=1,
+      two=2,
+      three="3"  # Invalid: Literal["3"] is not of type int
+   )
+
+In spite of the utility of this syntax, when trying to use it with a function, an error
+is raised, as functions are not subscriptable.
+
+.. code-block:: python
+
+   def my_list[T](arr) -> list[T]:
+      # do something...
+      return list(arr)
+
+   my_integer_list = my_list[int]()  # TypeError: 'function' object is not subscriptable
+
+There are a few workarounds:
+
+1. Making a callable class:
 
 .. code-block:: python
 
-   def make_list[T](*args: T) -> list[T]: ...
-   reveal_type(make_list())  # type checker cannot infer a meaningful type for T
+   class my_list[T]:
+        def __call__(self, *args: T) -> list[T]:
+            # do something...
+            return list(args)
 
-Making instances of ``FunctionType`` subscriptable would allow for this constructor to
-be typed:
+2. Using :pep:`747`\'s TypeForm, with an extra unused argument:
 
 .. code-block:: python
 
-   reveal_type(make_list[int]())  # type is list[int]
+   from typing import TypeForm
+
+   def my_list(*args: T, typ: TypeForm[T]) -> list[T]:
+      # do something...
+      return list(args)
+
+As we can see this solution increases the complexity with an extra argument.
+Additionally it requires the user to understand a new concept ``TypeForm``.
 
-Currently you have to use an assignment to provide a precise type:
+3. Annotating the assignment:
 
 .. code-block:: python
 
-   x: list[int] = make_list()
-   reveal_type(x)  # type is list[int]
+   my_integer_list: list[int] = my_list()
+
+This solution isn't optimal as the return type is repeated and is more verbose and
+would require the type updating in multiple places if the return type changes.
+
+In conclusion, the current workarounds are too complex or verbose, especially compared
+to syntax that is consistent with the rest of the language.
+
+Generic Specialisation
+^^^^^^^^^^^^^^^^^^^^^^
+
+As in the previous example currently we can create generic aliases for different
+specialised usages:
+
+.. code-block:: python
+
+   NameNumberDict = dict[str, int]
+   NameNumberDict(one=1, two=2, three="3")  # Invalid: Literal["3"] is not of type int``
+
+This not currently possible for functions but if allowed we could easily
+specialise operations in certain sections of the codebase:
+
+.. code-block:: python
+
+   def constrained_addition[T](a: T, b: T) -> T: ...
+
+   # where we work exclusively with ints
+   int_addition = constrained_addition[int]
+   int_addition(2, 4+8j)  # Invalid: complex is not of type int
+
+Unknown Types
+^^^^^^^^^^^^^
 
-but this code is unnecessarily verbose taking up multiple lines for a simple function
-call.
+Currently, it is not possible to infer the type parameters to generic functions in
+certain situations.
 
-Similarly, ``T`` in this example cannot currently be meaningfully inferred, so ``x`` is
+In this example ``T`` cannot currently be meaningfully inferred, so ``x`` is
 untyped without an extra assignment:
 
 .. code-block:: python
@@ -66,11 +138,11 @@ If function objects were subscriptable, however, a more specific type could be g
 
    reveal_type(factory[int](lambda x: "Hello World" * x))  # type is Foo[int]
 
-Undecidable Inference
-^^^^^^^^^^^^^^^^^^^^^
+Undecidable Inference and Type Narrowing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-There are even cases where subclass relations make type inference impossible. However,
-if you can specialise the function type checkers can infer a meaningful type.
+There are cases where subclass relations make type inference impossible. However, if
+you can specialise the function type checkers can infer a meaningful type.
 
 .. code-block:: python
 
@@ -138,7 +210,16 @@ The syntax for such a feature may look something like:
 Rationale
 ---------
 
-Function objects in this PEP is used to refer to ``FunctionType``\ , ``MethodType``\ ,
+This proposal improves the consistency of the type system, by allowing syntax that
+already looks and feels like a natural of the existing syntax for classes.
+
+If accepted, this syntax will reduce the necessity to learn about :pep:`747`\s
+``TypeForm``,  reduce verbosity and cognitive load of safely typed python.
+
+Specification
+-------------
+
+In this PEP "Function objects" is used to refer to ``FunctionType``\ , ``MethodType``\ ,
 ``BuiltinFunctionType``\ , ``BuiltinMethodType`` and ``MethodWrapperType``\ .
 
 For ``MethodType`` you should be able to write:
@@ -161,9 +242,6 @@ functions implemented in Python as possible.
 ``MethodWrapperType`` (e.g. the type of ``object().__str__``) is useful for
 generic magic methods.
 
-Specification
--------------
-
 Function objects should implement ``__getitem__`` to allow for subscription at runtime
 and return an instance of ``types.GenericAlias`` with ``__origin__`` set as the
 callable and ``__args__`` as the types passed.
@@ -201,10 +279,68 @@ The following code snippet would fail at runtime without this change as
 Interactions with ``@typing.overload``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Overloaded functions should work much the same as already, since they have no effect on
-the runtime type. The only change is that more situations will be decidable and the
-behaviour/overload can be specified by the developer rather than leaving it to ordering
-of overloads/unions.
+This PEP opens the door to overloading based on type variables:
+
+.. code-block:: python
+
+   @overload
+   def serializer_for[T: str]() -> StringSerializer: ...
+   @overload
+   def serializer_for[T: list]() -> ListSerializer: ...
+
+   def serializer_for():
+       ...
+
+For overload resolution a new step will be required previous to any other, where the resolver
+will match only the overloads where the subscription may succeed.
+
+.. code-block:: python
+
+   @overload
+   def make[*Ts]() -> float: ...
+   @overload
+   def make[T]() -> int: ...
+
+   make[int]  # matches first and second overload
+   make[int, str]  # matches only first
+
+
+Functions Parameterized by ``TypeVarTuple``\ s
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Currently, type checkers disallow the use of multiple ``TypeVarTuple``\s in their
+generic parameters; however, it is currently valid to have a function as such:
+
+.. code-block:: python
+
+   def foo[*T, *U](bar: Bar[*T], baz: Baz[*U]): ...
+   def spam[*T](bar: Bar[*T]): ...
+
+This PEP does not allow functions like ``foo`` to be subscripted, for the same reason
+as defined in :pep:`PEP 646<646#multiple-type-variable-tuples-not-allowed>`.
+
+.. code-block:: python
+
+   foo[int, str, bool, complex](Bar(), Baz())  # Invalid: cannot determine which parameters are passed to *T and *U. Explicitly parameterise the instances individually
+   spam[int, str, bool, complex](Bar())        # OK
+
+Binding Rules
+^^^^^^^^^^^^^
+Method subscription (including ``classmethods``, ``staticmethods``, etc.) should only
+have access to their function's type parameter and not the enclosing class's.
+Subscription should follow the rules specified in :pep:`PEP 696<696#binding-rules>`;
+methods should bind type parameters on attribute access.
+
+.. code-block:: python
+
+   class C[T]:
+       def method[U](self, x: T, y: U): ...
+       @classmethod
+       def cls[U](cls, x: T, y: U): ...
+
+   C[int].method[str](0, "")  # OK
+   C[int].cls[str](0, "")     # OK
+   C.cls[int, str](0, "")     # Invalid: too many type parameters
+   C.cls[str](0, "")          # OK, T is ideally bound to int here though this is open for type checkers to decide
 
 Backwards Compatibility
 -----------------------