Allow variadic generics

C++11 recently introduced the notion of variadic templates, which I believe Python could benefit from in a simplified form.

The idea is that you can have a generic class with a variable number of type variables, like typing.Tuple[] has. Here is a real-world variadic-generic class which is not Tuple; it is variadic in TagVar. As you can see, TagVar only appears in tuple contexts. Those tuples are sometimes heterogenous (Caution: annotations are a homegrown 3.4-compatible mishmash of nonsense), so repeating TagVar as shown is actually incorrect (but the closest approximation I could find).

Here's one possible syntax:

class MultiField(AbstractField[GetSetVar], Generic[(*TagVar,)]):
    def __init__(self, nbt_names: ty.Sequence[str], *, default:
                 GetSetVar=None) -> None:
        ...

    @abc.abstractmethod
    def to_python(self, *tags: (*TagVar,)) -> GetSetVar:
        ...

    @abc.abstractmethod
    def from_python(self, value: GetSetVar) -> ty.Tuple[(*TagVar,)]:
        ...

This is syntactically valid in Python 3.5 (if a bit ugly with the parentheses and trailing comma, which cannot be omitted without language changes), but doesn't currently work because type variables are not sequences and cannot be unpacked. It could be implemented by adding something like this to the TypeVar class:

def __iter__(self):
    yield StarredTypeVar(self)

StarredTypeVar would be a wrapper class that prefixes the repr with a star and delegates all other functionality to the wrapped TypeVar.

Of course, syntax isn't everything; I'd be fine with any syntax that lets me do this. The other immediately obvious syntax is to follow the TypeVar with an ellipsis, which conveniently does not require changes to typing.py. However, that might require disambiguation in some contexts (particularly since Tuple is likely to be involved with these classes).

Hmm... why not make this a special property of type variables.Tthen you wouldn't need the funny syntax, you could just use a type variable that has a keyword saying it is variadic.

But maybe the bigger question is how exactly we should type check this.

Hmm... why not make this a special property of type variables.Tthen you wouldn't need the funny syntax, you could just use a type variable that has a keyword saying it is variadic.

This isn't obviously objectionable, but for the following reasons I'm not sure it's actually a good idea. First, I would like to point out that I failed to notice a third possible syntax, which doesn't require the trailing comma:

class MultiField(AbstractField[GetSetVar], Generic[[*TagVar]]):
    def __init__(self, nbt_names: ty.Sequence[str], *, default:
                 GetSetVar=None) -> None:
        ...

    @abc.abstractmethod
    def to_python(self, *tags: [*TagVar]) -> GetSetVar:
        ...

    @abc.abstractmethod
    def from_python(self, value: GetSetVar) -> ty.Tuple[[*TagVar]]:
        ...

I'll return to this syntax in a moment.

But maybe the bigger question is how exactly we should type check this.

We should enforce that the variable only appears as the sole argument to Tuple (with a trailing ellipsis?), as a variable of a generic class, possibly (?) in the instantiation of a generic class other than Tuple, or as the type of a variadic parameter (*args, **kwargs). The generic class case is complicated, however, because there might be multiple variadic type variables in play under multiple inheritance. Even if there aren't, it could be difficult to parse.

We can overcome both problems by requiring specialized syntax when the generic class is subclassed or instantiated:

class UUIDField(MultiField[uuid.UUID, [tags.LongTag, tags.LongTag]]):
    ...

Note the extra pair of brackets. They indicate where the variadic typevar begins and ends, which removes any ambiguity when there are multiple variadic typevars, and simplifies parsing when there are non-variadic typevars in the same class (as in this case).

For reasons of uniformity, I would recommend we use the bracket syntax I showed above rather than making variadic-ness a property of the type variable. That makes instantiation and declaration look more like one another, and seems more intuitive to me.

Variadic generics would likely be useful at least occasionally. It would make sense to generalize them also to argument types in Callable. I remember that there are a few places in the std lib stubs where these would have let us have a more precise type for a library function. The canonical example might be something like this (in my counter-proposal being variadic is a property of a type variable, as it seems more readable to me):

Ts = TypeVar('Ts', variadic=True)
RT = TypeVar('RT')

def call_async(function: Callable[Ts, RT], args: Tuple[Ts]) -> Future[RT]:
    ...

In my proposal a variadic type variable Ts would be valid in at least these contexts:

As Tuple[Ts] (the only argument to Tuple)
As Callable[Ts, X] (the first argument to Callable)
As type of *args, such as *args: Ts

It's less obvious how to generalize this to user-defined generic types. Maybe like this:

class C(Generic[Ts]):
    def __init__(self, *x: Ts) -> None: ...

c = C(1, '')  # type: C[int, str]   # ok!

The limitation would be that in order to have both variadic and non-variadic arguments, you'd have to create a dummy wrapper type for the variadic args:

class ArgTypes(Generic[Ts]): pass   # dummy wrapper type

class C(Generic[T, ArgTypes[Ts]]):
    def __init__(self, t: T, *args: Ts) -> None: ...

The apparent awkwardness of the above example probably wouldn't matter much as I expect this use case to be very rare.

There are additional implementation details that a type checker should probably get right to make this useful, but I'm not even trying to enumerate them here. The implementation would likely be tricky, but probably not excessively so.

I'm still unconvinced that this is a very important feature -- look at how long it took C++ to pick up this feature. I'd propose looking for more use cases to justify variadic generics and once we have a sufficient number of use cases collected here, we'd send this to python-ideas@ for discussion.

Examples where these could be useful:

zip
contextlib.contextmanager

@JukkaL

@JukkaL There's one extension to your proposal that would solve a problem I've been trying to figure out: I have some Query objects, which vary in the types of results they produce. I also have an execute function, which takes in some Querys and would like to return a tuple, with one result per Query it was passed.

R = TypeVar('R')
class Query(Generic[R]):
   ...

Rs = TypeVar('Rs', variadic=True)
Qs = Rs.map(Query) 

def execute(*args: Qs) -> Tuple[Rs]:
   ...

I am not at all wed to this syntax for mapping over variadic types variables, but I want to be able to do it, and I think it should be a thing we should consider when considering variadic type variables.

It's not very clear from that example that execute() returns a tuple. I think Jukka's proposal would have you write Tuple[Rs] for the return type. I also wish the map() functionality was expressible as part of the signature of execute() rather than in the definition of Qs.

Anyway, this would be shorthand for an infinite sequence of definitions for execute(), like this, right?

R = TypeVar('R')
R1 = TypeVar('R1')
R2 = TypeVar('R2')
R3 = TypeVar('R3')
# etc. until R999
@overload
def execute(q1: Query[R]) -> R: ...
@overload
def execute(q1: Query[R1], q2: Query[R2]) -> Tuple[R1, R2]: ...
@overload
def execute(q1: Query[R1], q2: Query[R2], q3: Query[R3]) -> Tuple[R1, R2, R3]: ...
# etc. until a total of 999 variants

NOTE: The special case for the first overload is not part of the special semantics for variadic type variables; instead there could be two overload variants, one for a single query, one for 2 or more.

Yes, execute returns a Tuple[Rs] and I simply was posting too quickly to check my work, apologies.

And yes, this would be exactly that infinite series of overloads.

@NYKevin

Making the type mapping in the argument list would be nice, but requires some care as to exactly where you are parameterizing a type over each element of your variadic type variable, and where you are parameterizing a type using all elements of your variadic type variable.

For example, I'd love to be able to write Query[Rs] and mean "a variadic type variable that is a query type for each result type in Rs" but then I want to also be able to write something like Tuple[Rs] meaning "a tuple of every return type in Rs".

One possibility is, borrowing some stars and parens from @NYKevin to have, I think, somewhat of a different meaning:

def execute(*args: Query[Rs]) -> Tuple[(*Rs,)]:
    ...

So maybe the notation ought to reflect that, and we should be able to write

Rn = TypeVar('Rn', variadic=True)
def execute(*q1: Query[Rn.one]) -> Tuple[Rn.all]: ...

Where .one and .all try to give hints on how to expand these.

Ooh, I like not having it be some kind of obtuse operator but rather english words. Consider each as a possibility to mean what you're using one for above?

def execute_tuple(queries: Tuple[Query[Rs.each].all]) -> Tuple[Rs.all]: ...

Unless, say, .all is the default, and you don't have to say it, and you have to say .each at every layer you mean to map the types along.

Here's "all is the default, you have to specify each"

def execute_tuple(queries: Tuple[Query[Rs.each]]) -> Tuple[Rs]: ...

Here's "each is the default, you have to specify all"

def execute_tuple(queries: Tuple[Query[Rs].all]) -> Tuple[Rs.all]: ...

I need to think about more examples. But note that we're talking about *queries...

…

--Guido (mobile)

Hi all. Thank you for mypy!

So, what is the plan going forward regarding variadic generics?

webhooks: Update comment about typing the webhook decorator.

Delete with_cursor decorator

@NYKevin

A few of us have actually been working on a draft of a PEP for variadics generics since last year, PEP 646. Eric Traut has very kindly contributed an initial implementation in Pyright, Pradeep Kumar Srinivasan has been working on an implementation in Pyre, and we're working on the additions to typing.py in this pull request. Sorry for the late notice in this thread; I do remember seeing this thread a long time ago but apparently it didn't stick in my memory.

@NYKevin As of the current draft, I think your use case with MultiField would look like this using the current PEP draft:

GetSetVar = TypeVar('GetSetVar')
TagVar = TypeVarTuple('TagVar')

class MultiField(AbstractField[GetSetVar], Generic[*TagVar]):
    def __init__(self, nbt_names: Sequence[str], *, default: GetSetVar = None) -> None:
        ...

    @abc.abstractmethod
    def to_python(self, *tags: *TagVar) -> GetSetVar:
        ...

    @abc.abstractmethod
    def from_python(self, value: GetSetVar) -> Tuple[*TagVar]:
        ...

@sixolet I think you've seen the thread in typing-sig about 646, but for the sake of other people reading this thread: the Query use case isn't supported by the current PEP; it used to be, but the PEP was getting too big, so we postponed this for a future PEP. The section we cut out is in this doc. Here's what the Query case would look like if we did use the proposal in that doc: (though of course there's still plenty of room for more discussion on this; there are some interesting proposals in this thread!)

R = TypeVar('R')
Rs = TypeVarTuple('Rs')

class Query(Generic[R]):
    ...

def execute(*args: *Map[Query, Rs]) -> Tuple[*Rs]:
    ...

@seaders I think your use case should be possible with:

Ts = TypeVarTuple('Ts')

@classmethod
def qwith_entities(cls, models: Tuple[*Ts]) -> List[Tuple[*Ts]]:
    return cls.query.with_entities(*models)

In any case, if you have any feedback on the current draft of the PEP, please do drop by the thread in typing-sig and leave us a message. Cheers!

Does TypeVarTuple support distribution inside other types (like dict) ?
For example: (*args: *dict[str, Ts]) -> dict[str, tuple[*Ts]] ?

I'm trying to annotate the following function, but I'm not sure this is supported. It's unclear from the PEP as there's no example.

_KeyT = TypeVar('_KeyT')
_ValuesT = TypeVarTuple('_ValuesT')


def zip_dict(*dicts: *dict[_KeyT, _ValuesT]) -> Iterable[_KeyT, tuple[*_ValuesT]]:
  """Iterate over items of dictionaries grouped by their keys."""
  for key in set(itertools.chain(*dicts)):
    yield key, tuple(d[key] for d in dicts)

@Conchylicultor

@Conchylicultor this is unsupported. There's been talk of adding a typing.Map operator that would support something like this, but I don't think there's any concrete progress yet.

Closing this issue as PEP 646 has been accepted and implemented (thanks @mrahtz and @pradeep90!).

webhooks: Update comment about typing the webhook decorator.

Navigation Menu

Repository navigation

Allow variadic generics #193

42 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Allow variadic generics #193

Description

Activity

gvanrossum commented on Mar 24, 2016

NYKevin commented on Mar 24, 2016

JukkaL commented on Mar 26, 2016

JukkaL commented on Apr 5, 2016

sixolet commented on Jul 25, 2016

gvanrossum commented on Jul 25, 2016

sixolet commented on Jul 25, 2016

sixolet commented on Jul 25, 2016

sixolet commented on Jul 26, 2016

gvanrossum commented on Jul 26, 2016

sixolet commented on Jul 26, 2016

sixolet commented on Jul 26, 2016

sixolet commented on Jul 26, 2016

gvanrossum commented on Jul 26, 2016

42 remaining items

dnaaun commented on Jul 27, 2020

mrahtz commented on Feb 20, 2021

Conchylicultor commented on Mar 15, 2022

JelleZijlstra commented on Mar 15, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions