Crash on latest Django Project with dmypy #11456

Activity

1. Thorbenl

   added

   bugmypy got something wrongmypy got something wrong

   on Nov 4, 2021
2. sobolevn commented on Nov 4, 2021

   

   sobolevn

   on Nov 4, 2021

   Last edited by sobolevn

   Member

   More actions

   Please, report it in our repo: https://github.com/typeddjango/django-stubs/

   Also:

   1. Minimal repro will help a lot!
   2. Try 1.9.0, it might help!

   Thanks!
3. Thorbenl commented on Nov 8, 2021

   

   Thorbenl

   on Nov 8, 2021

   Author

   More actions

   @sobolevn I will try. Sadly, could only reproduce it on a fresh repo ones, and then it went away. For our project tho, it never goes away, does not matter which version combination i would take with stubs and mypy :( Ima try to pick it out of our giant repo.

   👍React with 👍1Reacted by sobolevn
4. christianbundy commented on Nov 16, 2021

   

   christianbundy

   on Nov 16, 2021

   Contributor

   More actions

   Having the same problem here, it started with a Pipfile.lock update recently and only affects dmypy. We're running django-stubs@1.9.0.
5. christianbundy commented on Nov 17, 2021

   

   christianbundy

   on Nov 17, 2021

   Contributor

   More actions

   @Thorbenl Any chance you have aiohttp as a dependency? I dropped into pdb and here's what I saw:

   (Pdb) lvalue.node.info
   <TypeInfo aiohttp.http_parser.HttpParser>

   After bisecting the project history, it looks like aio-libs/aiohttp@a6c7f15 as the problematic commit.

   You can repro by creating mypy.ini in aiohttp with the contents:

   [mypy]
   cache_fine_grained = True
   ignore_errors = True

   And running mypy ..

   (Technically you don't have to run errors, but this lets you bisect automatically.)

   cc: @asvetlov
6. christianbundy commented on Nov 17, 2021

   

   christianbundy

   on Nov 17, 2021

   Contributor

   More actions

   Found the problem, although I'm not completely sure why it's a problem:

   - class HttpParser(abc.ABC):
   + class HttpParser(abc.ABC, Generic[_MsgT]):
7. christianbundy commented on Nov 17, 2021

   

   christianbundy

   on Nov 17, 2021

   Contributor

   More actions

   Full traceback:

   aiohttp/http_parser.py: error: INTERNAL ERROR -- Please try using mypy master on Github:
   https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
   Please report a bug at https://github.com/python/mypy/issues
   version: 0.920+dev.43eecdb0969c8cb8777fa802ba53e11075b4c802
   Traceback (most recent call last):
     File "/usr/local/bin/mypy", line 8, in <module>
       sys.exit(console_entry())
     File "/usr/local/lib/python3.9/site-packages/mypy/__main__.py", line 11, in console_entry
       main(None, sys.stdout, sys.stderr)
     File "/usr/local/lib/python3.9/site-packages/mypy/main.py", line 92, in main
       res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
     File "/usr/local/lib/python3.9/site-packages/mypy/main.py", line 170, in run_build
       res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
     File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 180, in build
       result = _build(
     File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 256, in _build
       graph = dispatch(sources, manager, stdout)
     File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2712, in dispatch
       process_graph(graph, manager)
     File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 3043, in process_graph
       process_stale_scc(graph, scc, manager)
     File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 3152, in process_stale_scc
       graph[id].finish_passes()
     File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2225, in finish_passes
       free_tree(self.tree)
     File "/usr/local/Cellar/python@3.9/3.9.8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
       self.gen.throw(typ, value, traceback)
     File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 1964, in wrap_context
       yield
     File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2222, in finish_passes
       self.update_fine_grained_deps(self.manager.fg_deps)
     File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2272, in update_fine_grained_deps
       merge_dependencies(self.compute_fine_grained_deps(), deps)
     File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2263, in compute_fine_grained_deps
       return get_dependencies(target=self.tree,
     File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 117, in get_dependencies
       target.accept(visitor)
     File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 326, in accept
       return visitor.visit_mypy_file(self)
     File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 177, in visit_mypy_file
       super().visit_mypy_file(o)
     File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 37, in visit_mypy_file
       d.accept(self)
     File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 1000, in accept
       return visitor.visit_class_def(self)
     File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 238, in visit_class_def
       super().visit_class_def(o)
     File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 73, in visit_class_def
       o.defs.accept(self)
     File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 1069, in accept
       return visitor.visit_block(self)
     File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 348, in visit_block
       super().visit_block(o)
     File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 41, in visit_block
       s.accept(self)
     File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 727, in accept
       return visitor.visit_func_def(self)
     File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 201, in visit_func_def
       super().visit_func_def(ex)
     File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 56, in visit_func_def
       self.visit_func(o)
     File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 53, in visit_func
       o.body.accept(self)
     File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 1069, in accept
       return visitor.visit_block(self)
     File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 348, in visit_block
       super().visit_block(o)
     File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 41, in visit_block
       s.accept(self)
     File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 1135, in accept
       return visitor.visit_assignment_stmt(self)
     File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 400, in visit_assignment_stmt
       self.process_lvalue(lvalue)
     File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 460, in process_lvalue
       lvalue_type = self.get_non_partial_lvalue_type(lvalue)
     File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 492, in get_non_partial_lvalue_type
       assert not lvalue.is_new_def
   AssertionError: 
   
8. christianbundy commented on Nov 17, 2021

   

   christianbundy

   on Nov 17, 2021

   Contributor

   More actions

   Minimal repro!

   from typing import Generic, TypeVar
   
   T = TypeVar('T', int, str)
   
   class ExampleClass(Generic[T]):
       def __init__(
           self
       ) -> None:
           self.example_attribute = None

   This seems to happen when:

   • Fine-grained cache is enabled (e.g. manually in config, or mandatory if using dmypy)
   • Some TypeVar (T) has 2+ types.
   • Some class subclasses Generic[T]
   • The init method on that class sets an attribute to None.

   🤯
9. sobolevn commented on Nov 17, 2021

   

   sobolevn

   on Nov 17, 2021

   Member

   More actions

   Btw, this works fine:

   from typing import Generic, TypeVar
   
   T = TypeVar('T', int, str)
   
   class ExampleClass(Generic[T]):
       def __init__(
           self
       ) -> None:
           ...

   The problem reveals itself only if self.example_attribute = None is in place.

   Great insight, thanks!

   👍React with 👍1Reacted by Christian Bundy
10. christianbundy commented on Nov 17, 2021

    

    christianbundy

    on Nov 17, 2021

    Contributor

    More actions

    Any ideas on how we might solve it? I haven't contributed to mypy, and honestly I'm not completely sure what purpose this assert has in the first place.
11. sobolevn commented on Nov 17, 2021

    

    sobolevn

    on Nov 17, 2021

    Member

    More actions

    It happens here:

     File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 400, in visit_assignment_stmt
        self.process_lvalue(lvalue)
      File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 460, in process_lvalue
        lvalue_type = self.get_non_partial_lvalue_type(lvalue)
      File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 492, in get_non_partial_lvalue_type
        assert not lvalue.is_new_def
    

    I've never touched this part of mypy before. And these lines of code are quite old (4 years):

    mypy/mypy/server/deps.py

    Lines 487 to 493 in 43eecdb

    	# Probably a secondary, non-definition assignment that doesn't
    	# result in a non-partial type. We won't be able to infer any
    	# dependencies from this so just return something. (The first,
    	# definition assignment with a partial type is handled
    	# differently, in the semantic analyzer.)
    	assert not lvalue.is_new_def
    	return UninhabitedType()

    Git Blame Link

    This can either be:

    1. A trivial if bug, where we just forgot to add some condition
    2. Quite complex one, due to our way-overly-compicated-plugin
12. sobolevn commented on Nov 17, 2021

    

    sobolevn

    on Nov 17, 2021

    Member

    More actions

    @christianbundy this seems to remove the crash:

                if isinstance(lvalue.node, Var):
                    if lvalue.node.type:
                        lvalue_type = get_proper_type(lvalue.node.type)
                    else:
                        lvalue_type = NoneType()

    But, I am not sure it is correct though. Do you want to send a PR? We can work from there!
13. christianbundy

    added a commit that references this issue on Nov 17, 2021

    Fix `None` assignments with fine-grained cache

    ...

    a4f7274
14. christianbundy commented on Nov 17, 2021

    

    christianbundy

    on Nov 17, 2021

    Contributor

    More actions

    Good idea, thank you!
15. hauntsaninja

    closed this as completedin #11574on Dec 8, 2021
16. hauntsaninja

    added a commit that references this issue on Dec 8, 2021

    Fix `None` assignments with fine-grained cache (#11574)

    ...

    1393e3f
17. severo

    added a commit that references this issue on Dec 14, 2021

    chore: 🤖 avoid an error in vsscode with mypy daemon

    ...

    348a6ae
18. tushar-deepsource

    added a commit that references this issue on Jan 20, 2022

    Fix `None` assignments with fine-grained cache (python#11574)

    ...

    d37ffcd
19. mattstern31

    added a commit that references this issue on Nov 11, 2023

    chore: 🤖 avoid an error in vsscode with mypy daemon

    ...

    0d0e4a5