Repository navigation
mypy not finding modules located above the current working directory #6660
Description
I have a project with a structure similar to:
- project
- mypy.ini
- src
- main.py
- pkg_a
- __init__.py
- mod_a.py
- pkg_b
- __init__.py
- mod_b.py
Where main.py has from pkg_a.mod_a import ThingFromA and mod_a has from pkg_b.mod_b import ThingFromB
If I run MyPy from project or from src, either pointing it at main.py or at pkg_a/mod_a.py, it works fine. If, however, I cd into pkg_a and run it on ./mod_a.py, it fails to resolve the import to package B.
The MyPy docs state that the behavior of absolute import resolution is to walk up the file tree until the first folder that does not have an __init__.py and treat that as the root of the project by adding it to the search path. It seems that there is an additional restriction here that MyPy will only walk as far up as the cwd it is called from. The docs make no mention of this that I can find. Is this an intentional limitation? I can reproduce the problem on a colleague's computer, but hours of googling have not managed to find anyone else complaining about this so I feel like I'm missing something obvious here.
The reason this is an issue is because, while I use VSCode, the team coming onto this project uses PyCharm on Windows, and the MyPy plugin we found for it seems to be calling MyPy from the directory of the file it is checking, meaning PyCharm users are currently stuck without working editor integration (we don't want to use PyCharm's own type checker as we want the type checking on our development machines to behave the same as the type checking in CI (which will be MyPy)).
I am able to reproduce this with the latest MyPy 0.7. The project is Python 3.6.
Activity
ilevkivskyi commentedon Apr 11, 2019
Is it also a problem with the official mypy PyCharm plugin?
ada-waffles commentedon Apr 11, 2019
My understanding was that that plugin doesn't work on Windows (based on this open issue). I can give it a try anyway...
ilevkivskyi commentedon Apr 11, 2019
Ah, yes sorry, it might not work on Windows indeed.
ada-waffles commentedon Apr 11, 2019
Yeah we tried that plugin and it does not, in fact, work on Windows (it's looking for
/bin/bash).Do you know if the behavior I describe with module resolution is correct? Could there be something wrong with how we're configuring MyPy or structuring our project?
gvanrossum commentedon Apr 11, 2019
What if you used an absolute path? I.e. (assuming you're inside
pkg_a)mypy C:\path\to\project\src\pkg_a\mod_a.py? Does that work or not?ada-waffles commentedon Apr 12, 2019
That does work! So it looks like it's not the cwd that's the issue, but the way the file is specified.
ada-waffles commentedon Apr 12, 2019
Took my best stab at a bit of code spelunking now that I was armed with that extra bit of information.
It looks to my completely-unfamiliar-with-your-codebase eye that the crawling is being done using string manipulation on the path, and that the behavior I'm seeing is being caused by the use of
os.path.normpathat https://github.com/python/mypy/blob/master/mypy/find_sources.py#L36, rather thanabspath. I'm going to guess that this was an intentional choice. Might I ask the reasoning?ada-waffles commentedon Apr 12, 2019
Looks like I've found two separate issues here, after all.
Turns out that PyCharm plugin is passing
--follow-imports skipfor some reason.I still find the behavior described in this issue odd, so I'll leave it open until someone at least confirms the behavior is intentional.
gvanrossum commentedon Apr 12, 2019
Good find. I don't actually know why it is done that way, but I can reproduce that if we replace that with abspath() it does find the sibling module. However I note that asking Python to execute that module (from inside the package) doesn't find the sibling module either. And mypy in general tries to emulate the import behavior of Python (with some exceptions). So this feels like it's intentional.
ada-waffles commentedon Apr 12, 2019
That is true. I guess the conflict of assumptions here then would be that I was assuming that calling MyPy on a file doesn't necessarily imply that that file is intended as an entry point, just that I want to type-check that file. I could totally understand the entry point assumption, though. I would, however, think that the documentation should be modified to reflect it if that assumption is taken as the canonical one. Right now the import discovery process is not described as having any such restriction.
gvanrossum commentedon Apr 12, 2019
ada-waffles commentedon Apr 12, 2019
I would be happy to. I'd want guidance on how exactly to write the clarification, though.
Would it be sufficient to adjust https://mypy.readthedocs.io/en/latest/running_mypy.html#how-imports-are-found by following the sentence
with
?
I confirmed the above description by calling MyPy on "main.py" (no .\) from src. It was still able to resolve the imports from main, so it does consider the cwd even if it's not explicitly a segment of the given path.
gvanrossum commentedon Apr 12, 2019
[-]MyPy not finding modules located above the current working directory[/-][+]mypy not finding modules located above the current working directory[/+]Add note about relative file paths in import discovery (#6676)