[0.530] AssertionError: Internal error: Unresolved forward reference to UnboundType

Traceback (most recent call last):
  File "/Users/daenyth/Curata/cmp/.tox/mypy/bin/mypy", line 11, in <module>
    sys.exit(console_entry())
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/main.py", line 103, in type_check_only
    options=options)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 198, in build
    graph = dispatch(sources, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 1841, in dispatch
    process_graph(graph, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 2091, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 2203, in process_stale_scc
    graph[id].write_cache()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 1822, in write_cache
    self.manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 1088, in write_cache
    data = tree.serialize()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 238, in serialize
    'names': self.names.serialize(self._fullname),
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2432, in serialize
    data[key] = value.serialize(fullname, key)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2371, in serialize
    data['node'] = self.node.serialize()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2194, in serialize
    'names': self.names.serialize(self.fullname()),
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2432, in serialize
    data[key] = value.serialize(fullname, key)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2371, in serialize
    data['node'] = self.node.serialize()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 533, in serialize
    'type': None if self.type is None else self.type.serialize(),
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 857, in serialize
    'arg_types': [t.serialize() for t in self.arg_types],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 857, in <listcomp>
    'arg_types': [t.serialize() for t in self.arg_types],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 496, in serialize
    data['args'] = [arg.serialize() for arg in self.args]
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 496, in <listcomp>
    data['args'] = [arg.serialize() for arg in self.args]
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 994, in serialize
    'items': [t.serialize() for t in self.items],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 994, in <listcomp>
    'items': [t.serialize() for t in self.items],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 1410, in serialize
    assert False, "Internal error: Unresolved forward reference to {}".format(name)
AssertionError: Internal error: Unresolved forward reference to UnboundType

I get this running on my codebase after updating to 0.530. It passes without error in 0.521. I'm not sure what's triggering it - if you can help me figure that out I can try to minimize the failure.

This is something related to a forward reference. Far example:

x: 'A'
A = List[int]

There is a chance (but not 100%) that you have a # type: ignore on the offending line.

Can you actually share the code? Alternatively I can prepare a "diagnostic" branch and then you could show me the output.

Unfortunately I can't share the code since it's closed source. I'll try to figure out which sub module is causing it and try to find a minimal reproduction.

The codebase is python2, if that helps.

Running with -v I get this immediately prior to the traceback:

LOG:  Processing the last 247 queued SCCs
LOG:  Processing SCC singleton (cmp.reporting.types) as inherently stale
LOG:  Writing cmp.reporting.types /Users/daenyth/Curata/cmp/cmp/reporting/types.py .mypy_cache/2.7/cmp/reporting/types.meta.json .mypy_cache/2.7/cmp/reporting/types.data.json
LOG:  Build finished in 7.030 seconds with 794 modules, 55 types, and 0 errors

I can confirm that running it on that single file fails, I'll try to minimize

It's something a little more complicated than just that.

The file contains a line similar to Foo = NewType('NotFoo', int). When I fix that, mypy does not crash.
When I copy the same file contents to a foo.py not inside the package directory on disk, mypy also does not crash.

The file contains a line similar to Foo = NewType('NotFoo', int)

This is a very good hint. I remember having problems with NewTypes I thought have cleaned-up them all, probably not.

Could you please grep for the places where Foo was used, it looks like some of this places is not visited in the third pass of semantic analysis.

It looks like it's used in multiple files.

The only direct use is as part of an alias for Bar = Tuple[Foo, Blah].
Bar is always used in a typevar position, eg List[Foo] or some other generic type
The files that it's used in have a mypy.ini section with ignore_errors = True

In particular, does it appear in an overload function signature? I have found a repro, but it is not related to NewTypes, it can happen with any broken forward reference in an overload.

The codebase doesn't have any uses of overload

OK, this means that we have at least two crashes :-)

Could you please try the original crash with mypy installed from current master? How does the crash look like? (There are chances it will crash differently.)

LOG:  Processing SCC singleton (cmp.reporting.types) as inherently stale with stale deps (__future__ builtins datetime typing)
LOG:  Writing cmp.reporting.types /Users/daenyth/Curata/cmp/cmp/reporting/types.py .mypy_cache/2.7/cmp/reporting/types.meta.json .mypy_cache/2.7/cmp/reporting/types.data.json
LOG:  Build finished in 0.972 seconds with 12 modules, 1757 types, and 0 errors
Traceback (most recent call last):
  File "/Users/daenyth/Curata/cmp/.tox/mypy/bin/mypy", line 11, in <module>
    load_entry_point('mypy===0.540-dev-77ce5e206089e796cf3c563b52791e8a0b4d2d54', 'console_scripts', 'mypy')()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/main.py", line 103, in type_check_only
    options=options)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 199, in build
    graph = dispatch(sources, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 1865, in dispatch
    process_graph(graph, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 2115, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 2227, in process_stale_scc
    graph[id].write_cache()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 1846, in write_cache
    self.manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 1090, in write_cache
    data = tree.serialize()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 238, in serialize
    'names': self.names.serialize(self._fullname),
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2440, in serialize
    data[key] = value.serialize(fullname, key)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2379, in serialize
    data['node'] = self.node.serialize()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2170, in serialize
    'names': self.names.serialize(self.fullname()),
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2440, in serialize
    data[key] = value.serialize(fullname, key)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 2379, in serialize
    data['node'] = self.node.serialize()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 510, in serialize
    'type': None if self.type is None else self.type.serialize(),
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 856, in serialize
    'arg_types': [t.serialize() for t in self.arg_types],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 856, in <listcomp>
    'arg_types': [t.serialize() for t in self.arg_types],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 496, in serialize
    data['args'] = [arg.serialize() for arg in self.args]
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 496, in <listcomp>
    data['args'] = [arg.serialize() for arg in self.args]
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 993, in serialize
    'items': [t.serialize() for t in self.items],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 993, in <listcomp>
    'items': [t.serialize() for t in self.items],
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 1418, in serialize
    assert False, "Internal error: Unresolved forward reference to {}".format(name)
AssertionError: Internal error: Unresolved forward reference to Foo

@Daenyth

I tried to play with this for some time, but still can't get a repro. It is quite mysterious, since the code provoking the crash looks very simple, from the traceback I see that there is a method (most likely in a module level class) that has one of the argument types of the form OneClass[Tuple[Foo, OtherClass]] where Foo is the broken NewType definition.

@Daenyth maybe you could find one of such methods, and isolate the related code, i.e. keep the import structure between files, but only leave the necessary definitions in the corresponding files, i.e. something like this:

# file a.py
from c import C
Foo = NewType('NotFoo', int)

# file b.py
from a import Foo
Bar = Tuple[Foo, int]

# file c.py
from b import Bar
class C:
    def meth(self, x):
        # type: (List[Bar]) -> None
        pass

And then see if this causes the crash? This will be very helpful.

OK, I was able to make a reproduction into a single file. The stack trace I get is not exactly the same as running on the real code, but it's a crash, and a similar message.

# This is python2
from __future__ import absolute_import, division, print_function, unicode_literals
from typing import Any, Dict, List, NewType

Foo = NewType('NotFoo', int)
Foos = NewType('Foos', List[Foo])

def frob_the_widget(foos):
    # type: (Dict[Any, Foos]) -> None
    f = foos.get(1)
    reveal_type(f)
    list(f)

Removing the list() invocation makes it no longer crash.

# mypy==0.530
bar.py:4: error: String argument 1 'NotFoo' to NewType(...) does not match variable name 'Foo'
bar.py:5: error: Invalid type "bar.Foo"
bar.py:10: error: Revealed type is 'Union[bar.Foos*, builtins.None]'
bar.py:11: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.530
Traceback (most recent call last):
  File "/Users/daenyth/Curata/cmp/.tox/mypy/bin/mypy", line 11, in <module>
    sys.exit(console_entry())
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/main.py", line 103, in type_check_only
    options=options)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 198, in build
    graph = dispatch(sources, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 1841, in dispatch
    process_graph(graph, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 2091, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 2194, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/build.py", line 1753, in type_check_first_pass
    self.type_checker.check_first_pass()
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checker.py", line 194, in check_first_pass
    self.accept(d)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checker.py", line 282, in accept
    stmt.accept(self)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 519, in accept
    return visitor.visit_func_def(self)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checker.py", line 530, in visit_func_def
    self.check_func_item(defn, name=defn.name())
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checker.py", line 590, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checker.py", line 750, in check_func_def
    self.accept(item.body)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checker.py", line 282, in accept
    stmt.accept(self)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 773, in accept
    return visitor.visit_block(self)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checker.py", line 1304, in visit_block
    self.accept(s)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checker.py", line 282, in accept
    stmt.accept(self)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/nodes.py", line 787, in accept
    return visitor.visit_expression_stmt(self)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checker.py", line 2001, in visit_expression_stmt
    self.expr_checker.accept(s.expr, allow_none_return=True, always_allow_any=True)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checkexpr.py", line 2368, in accept
    typ = self.visit_call_expr(node, allow_none_return=True)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checkexpr.py", line 266, in visit_call_expr
    ret_type = self.check_call_expr_with_callee_type(callee_type, e, fullname, object_type)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checkexpr.py", line 503, in check_call_expr_with_callee_type
    object_type=object_type)[0]
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checkexpr.py", line 608, in check_call
    object_type=object_type)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checkexpr.py", line 565, in check_call
    callee, args, arg_kinds, formal_to_actual, context)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/checkexpr.py", line 813, in infer_function_type_arguments
    strict=self.chk.in_checked_function())
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/infer.py", line 31, in infer_function_type_arguments
    callee_type, arg_types, arg_kinds, formal_to_actual)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/constraints.py", line 63, in infer_constraints_for_callable
    SUPERTYPE_OF)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/constraints.py", line 147, in infer_constraints
    res.extend(infer_constraints(template, a_item, direction))
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/constraints.py", line 174, in infer_constraints
    return template.accept(ConstraintBuilderVisitor(actual, direction))
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 476, in accept
    return visitor.visit_instance(self)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/constraints.py", line 332, in visit_instance
    mapped = map_instance_to_supertype(instance, template.type)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/maptype.py", line 23, in map_instance_to_supertype
    return map_instance_to_supertypes(instance, superclass)[0]
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/maptype.py", line 36, in map_instance_to_supertypes
    a.extend(map_instance_to_direct_supertypes(t, sup))
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/maptype.py", line 81, in map_instance_to_direct_supertypes
    t = expand_type(b, env)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/expandtype.py", line 16, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 476, in accept
    return visitor.visit_instance(self)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/expandtype.py", line 83, in visit_instance
    args = self.expand_types(t.args)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/expandtype.py", line 132, in expand_types
    a.append(t.accept(self))
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 1399, in accept
    return visitor.visit_forwardref_type(self)
  File "/Users/daenyth/Curata/cmp/.tox/mypy/lib/python3.6/site-packages/mypy/types.py", line 1489, in visit_forwardref_type
    raise RuntimeError('Internal error: unresolved forward reference')
RuntimeError: Internal error: unresolved forward reference
bar.py:11: : note: use --pdb to drop into pdb

Thank you! I will make a fix tomorrow and you can then test if it fixes the original crash (with all files).

At least I can reproduce the original traceback with slightly modified repro.

@Daenyth

@Daenyth Could you please check that PR #4120 fixes your original crash (in your private repo)?

Fix recent crashes in forward refs (#4120)

Navigation Menu

Repository navigation

[0.530] AssertionError: Internal error: Unresolved forward reference to UnboundType #4097

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[0.530] AssertionError: Internal error: Unresolved forward reference to UnboundType #4097

Description

Activity

ilevkivskyi commented on Oct 11, 2017

Daenyth commented on Oct 11, 2017

Daenyth commented on Oct 11, 2017

Daenyth commented on Oct 11, 2017

ilevkivskyi commented on Oct 11, 2017

ilevkivskyi commented on Oct 11, 2017

Daenyth commented on Oct 11, 2017

ilevkivskyi commented on Oct 11, 2017

Daenyth commented on Oct 11, 2017

ilevkivskyi commented on Oct 11, 2017

ilevkivskyi commented on Oct 11, 2017

Daenyth commented on Oct 11, 2017

ilevkivskyi commented on Oct 12, 2017

Daenyth commented on Oct 12, 2017

ilevkivskyi commented on Oct 12, 2017

ilevkivskyi commented on Oct 12, 2017

ilevkivskyi commented on Oct 13, 2017

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions