Type variable with NamedTuple as upper bound not working

I'm trying to type annotate a method that takes a NamedTuple subclass as an argument and returns an instance of that class. Here is a complete example:

from typing import Dict
from typing import NamedTuple
from typing import Type
from typing import TypeVar

Struct = TypeVar('Struct', bound=NamedTuple)

def namedtuple_from_dict(nt_class: Type[Struct], dict_values: Dict) -> Struct:
    """Create a namedtuple from a dict, using the namedtuple attribute names
     to look up values in the dict."""
    return nt_class._make(dict_values.get(k) for k in nt_class._fields)  # type: ignore

class Point(NamedTuple):
    x: int
    y: int

namedtuple_from_dict(Point, {'x': 5, 'y': 3})
# error: Type argument 1 of "namedtuple_from_dict" has incompatible value "Point"

I might be doing something wrong, but I think according to the documentation this should work. I've seen a couple of bugs being open related to NamedTuples, but not one that seemed to fit this case.

The type: ignore for the return statement is needed because otherwise mypy says the function returns a NamedTuple instead of a Struct instance - which seems to be another bug...?

With master the error is slightly different:

error: Value of type variable "Struct" of "namedtuple_from_dict" cannot be "Point"

As a work-around you could remove the bound=NamedTuple from the TypeVar() call, but I agree it's surprising this isn't accepted.

The point is that NamedTuple is not a real class, it is rather a special form. The same question appeared on typing tracker, see python/typing#431, a possible way proposed there is to define your own protocol that mimics collections.namedtuple interface, then you don't even need the # type: ignore:

from typing_extensions import Protocol

T = TypeVar('T')
class NTProto(Protocol):
    _source: str
    _fields: Tuple[str, str]
    @classmethod
    def _make(cls: Type[T], iterable: Iterable[Any]) -> T: ...
    # add other methods, if needed

NT = TypeVar('NT', bound=NTProto)
def namedtuple_from_dict(nt_class: Type[NT], dict_values: Dict[str, Any]) -> NT:
    return nt_class._make(dict_values.get(k) for k in nt_class._fields)

class Point(NamedTuple):
    x: int
    y: int

namedtuple_from_dict(Point, {'x': 5, 'y': 3})  # OK, return type is Point
namedtuple_from_dict(int, {})  # Fails

This is however only works on master (release is planned in few weeks, also for typing_extensions).

@gvanrossum

Thanks @gvanrossum, I'll use that workaround for now.

Good to know @ilevkivskyi, I'll be looking for the typing_extensions release and will try out your solution!

I have another strange example of this.

from typing import *

A = TypeVar('A', bound=NamedTuple)

class Foo(Generic[A]):
    pass

B = NamedTuple('B', [
    ('b', float),
])

class Bar(Foo[B]):
    pass

Error:

bar.py:12: error: Type argument "Tuple[builtins.float, fallback=bar.B]" of "Foo" must be a subtype of "typing.NamedTuple"

Python 3.5.2
mypy 0.521

@JukkaL

I am a bit surprised that several people asked about something like:

x: NamedTuple = Point(1, 2)

But there is actually a simple solution to this. NamedTuple can be declared as a protocol in typeshed, then everything will work. We can list the same protocol members that are added by mypy when creating a synthetic TypeInfo plus stuff from tuple, so that the full definition will be:

NT = TypeVar('NT', bound='NamedTuple')
class NamedTuple(tuple, Protocol):  # type: ignore # since 'tuple' is not a protocol
    _fileds: Tuple[str, ...]  # this should be fixed, but this is easy
    _field_types: Dict[str, Any]
    _field_defaults: Dict[str, Any]
    _source: str
    def _replace(self: NT) -> NT: ...
    def _asdict(self) -> Dict[str, Any]:
    @classmethod
    def _make(cls: Type[NT], iterable: Iterable[Any], *, new: Any = ..., len: Any = ...) -> NT: ...

@JukkaL what do you think?

To be honest that looks kind of ugly, since protocols aren't supposed to derive from concrete classes. Also, it isn't type safe since it's deriving from Tuple[Any, ...].

I think that the root of the problem is that NamedTuple isn't a real type, it's a type constructor. A better workaround would be to disallow using NamedTuple as a type -- it's surprising that mypy doesn't complain about it already. It's still possible to create a user-defined protocol with the NamedTuple members that are relevant.

it's surprising that mypy doesn't complain about it already

It is defined as a class in typeshed, therefore it is valid in annotations, but yes we can just easily prohibit this. Maybe it makes sense to mention this in common issues?

Maybe it makes sense to mention this in common issues?

Just rejecting this is good enough for now. If people still continue to ask about this in the future we can add it to common issues.

Why do people ask for x: NamedTuple = Point(1, 2) ? What do they think they would gain over x: tuple = Point(1, 2)?

Why do people ask for x: NamedTuple = Point(1, 2) ? What do they think they would gain over x: tuple = Point(1, 2)?

My understanding is that they want _fields, _make, and _asdict (judging from the original example in this issue and elsewhere).

In our case, we are defining an interface for others to use. So we would like them to define a type they will pass to us, because we would like to have (name, type, value). In concrete example, we have an API for machine learning primitives. They have internally parameters they learned (weights on deep learning layers, etc.). The usefulness of NamedTuple seems to be that they can define their own set of parameters for their primitive, name each value, put a type on it. So we have something like:

Params = TypeVar('Params', bound=NamedTuple):

class Primitive(Generic[Params]):
    def get_params(self) -> Params:
        ...

    def set_parmas(self, params: Parms) -> None:
        ...

One extra thing which we are looking into and seems also not easily possible is to define a second type, which would be based on the Params. So in machine learning these days everyone wants to compute gradients on their params. So what would be great is to define a type:

def get_gradients(self) -> Gradients[Params, float]:
    ...

In this case, the return type should be of same structure as Params, have the same fields, same field names, only that all types of those fields would be float.

The issue is that for example type Sequence[Tuple[str, Type, Any]] is not good for Params because we would like that it is required to specify all of parameters one defined. Also, different fields have different types.

@ilevkivskyi

@ilevkivskyi I just found this issue while trying to create a function that creates an instance of a named tuple based on its fields and a given dict:

from typing import NamedTuple, TypeVar, Type

T = TypeVar('T', bound=NamedTuple)

class MyClass(NamedTuple):
    ...

def from_dict(cls: Type[T]) -> T:
    ...

from_dict(MyClass)

I wanted to typehint that the parameter should be a subclass of NamedTuple but I hit the error error: Value of type variable "T" of "from_dict" cannot be "MyType". Maybe the issue is obvious for some people (it still isn't for me), but I still think it would be nice to add something to the docs about this, it would have saved me quite some time. :)

This keeps coming up. I think we should just allow NamedTuple (or classes derived from it) as an upper bound for type variables.

@sephii

As a stopgap solution we could at least mention this issue in the documentation, as suggested by @sephii. Using a user-defined protocol as the bound seems like a potential workaround.

Not sure if this should be a separate bug, or considered part of this one, but the issue is not specific to TypeVar bounds. Annotating an argument directly as type NamedTuple also does not work:

from typing import NamedTuple

class Person(NamedTuple):
    name: str

def foo(x: NamedTuple) -> None:
    pass

p = Person(name="Name")

foo(p)

Fails with: :11: error: Argument 1 to "foo" has incompatible type "Person"; expected "NamedTuple"

The general issue seems to be what Ivan mentioned, "NamedTuple is not a class, it's a special form"

Treating NamedTuple as a class would be problematic for a few technical reasons:

Named tuples don't have a specific common base class like NamedTuple in their MRO (other than tuple), and if mypy treated it as such a common base class, it would be inconsistent and potentially confusing (and even unsafe).
namedtuple and NamedTuple can't be used meaningfully as the second argument in isinstance checks, unlike classes. NamedTuple doesn't generate an error when used as the type in isinstance, but that seems like a bug since it doesn't actually do the right thing, as far as I can tell -- it just returns False for named tuple instances.

I'm actually starting to wonder if supporting NamedTuple as a base class was a great idea, since it's actually not a real base class. Maybe we should have used a class decorator instead. It's probably too late to change that, however.

If we want to support typing.NamedTuple as a type, it looks like it would have to be a new special type like Any, Tuple[...] and Callable instead of a regular class. The use case however doesn't seem important enough to add a totally new kind of type, since the required implementation complexity is considerable.

One potential compromise would be to have a predefined a protocol/ABC that can be used instead of NamedTuple for a similar effect. It would have to have a different name, but if a user tried to use NamedTuple as a type, mypy could suggest using the corresponding protocol/ABC instead. Not sure what to call this, though. We could add it to mypy_extensions as a first step if we agree that this would be a good thing to have.

Here are some random ideas for the name of the hypothetical protocol/ABC:

AbstractNamedTuple (like AbstractSet)
NamedTupleProtocol

The protocol would include all relevant tuple and named tuple methods. If we'd make it an ABC, it could subclass tuple.

I don't understand why people want to use NamedTuple as a TypeVar upper bound. There seems no advantage (from a type-checking point) over using tuple.

I think the main use case for me at least is a simple and quick factory for a class which should hold only few predefined and required attributes + some custom methods. I moved our code to use that instead of NamedTuples and it solved the problem.

I think NamedTuples could be made into something similar to Enums. Where are literally a custom class and are in MRO.

That sounds like you're answering the question "what's the use case for NamedTuple". What does your code look like, what does it do, that it needs a TypeVar with plain NamedTuple in the upper bound? I'd expect an answer of the form "if I don't put NamedTuple in the upper bound of the TypeVar, I wouldn't get a useful error message for broken code of the following form: ".

FWIW there is a difference between NamedTuple and Enum, in that the untyped stdlib uses namedtuple (all lowercase) and that doesn't show up in the MRO -- changing that would likely break code so is not an option. And we prefer NamedTuple to be as similar as possible to namedtuple so it's a safe substitution to make.

Maybe output a tailored error message: Did you mean "Tuple" instead of "NamedTuple"?

It would be nice if typing.NamedTuple had a verbose=True option (like collections.namedtuple), to show what's generated. And even better if it could generate a proper Tuple annotation to avoid the problem in the first place ...

[Of course, namedtuple should be replaced by something better, but that's a separate issue.]

As suggested by @sephii and @JukkaL, it would be useful to add something to the docs to clarify the relevant details of NamedTuple discussed in this and other related issues. I bumped into a problem similar to the one described by @sephii. In my case, @overload provided a nice workaround.

It seems as though mypy will now happily accept NamedTuple as the upper bound of a TypeVar, and as an annotation. These are the two major pain points discussed in this thread, so I'm closing this issue.

Navigation Menu

Repository navigation

Type variable with NamedTuple as upper bound not working #3915

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Type variable with NamedTuple as upper bound not working #3915

Description

Activity

gvanrossum commented on Sep 4, 2017

ilevkivskyi commented on Sep 4, 2017

sjaensch commented on Sep 6, 2017

mitar commented on Sep 17, 2017

ilevkivskyi commented on Sep 17, 2017

JukkaL commented on Sep 17, 2017

ilevkivskyi commented on Sep 17, 2017

JukkaL commented on Sep 17, 2017

gvanrossum commented on Sep 17, 2017

ilevkivskyi commented on Sep 17, 2017

mitar commented on Sep 17, 2017

sephii commented on Feb 27, 2018

gvanrossum commented on Feb 27, 2018

JukkaL commented on Feb 27, 2018

carljm commented on Mar 10, 2018

JukkaL commented on Mar 12, 2018

gvanrossum commented on Mar 12, 2018

mitar commented on Mar 12, 2018

gvanrossum commented on Mar 12, 2018

kamahen commented on Mar 12, 2018

khyox commented on Jul 1, 2018

AlexWaygood commented on Mar 27, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions