Repository navigation
Discovery strategy is still mysterious w/PEP 420 namespace packages #6989
Description
I'm having trouble understanding some of the subtleties of how mypy 0.701 (w/Python 3.7) discovers files to check. Here is a sample directory tree:
.
└── lipy-foo
├── setup.cfg
└── src
└── linkedin
└── foo
├── __init__.py
└── bar.py
linkedin is a PEP 420 namespace package and linkedin/foo/__init__.py is empty. bar.py contains:
def bar(a: int, b: int) -> str:
return a + b
When I cd into the lipy-foo directory containing the setup.cfg, and try to run mypy, it can't find any .py files to check:
% ls
setup.cfg src/
% mypy src --namespace-packages -v
There are no .py[i] files in directory 'src'
Clearly mypy doesn't recurse into the src directory to find the linkedin namespace package. Other options that don't work include:
% mypy -p linkedin --namespace-packages
Can't find package 'linkedin'
% MYPYPATH=src mypy -p linkedin --namespace-packages
Can't find package 'linkedin'
What does work:
% mypy src/linkedin --namespace-packages
src/linkedin/foo/bar.py:2: error: Incompatible return value type (got "int", expected "str")
Also, if I add an empty src/linkedin/__init__.py file, discovery does work as expected:
% tree
.
├── setup.cfg
└── src
└── linkedin
├── __init__.py
└── foo
├── __init__.py
└── bar.py
% mypy src --namespace-packages --no-site-packages
src/linkedin/foo/bar.py:2: error: Incompatible return value type (got "int", expected "str")
So PEP 420 style namespace packages do seem to influence the behavior here.
Since we may have additional directories and namespace packages under src in other repositories, I'd prefer not to have to invoke mypy with src/linkedin, but there doesn't seem to be any other option.
Maybe I'm doing something wrong, so please tell me if this is pebkac, or is unsupported. Should mypy be able to discover the linkedin namespace package using src-style package layouts with PEP 420 namespaces?
Activity
ilevkivskyi commentedon Jun 14, 2019
I think @gvanrossum is the best person to ask about this.
gvanrossum commentedon Jun 14, 2019
This is totally mypy's fault. Namespace packages are only discovered during import following, not during initial file discovery from the command line. I recommend dropping an empty
__init__.pyi(note:.pyi) in the namespace package.warsaw commentedon Jun 14, 2019
Thanks for the follow up Guido. Adding an empty placeholder file isn't really an option for us, for reasons not worth going into here.
Would it make sense, when the
-poption is given, to do the import and use the resulting module's__spec__.submodule_search_locationsto find the directory to start scanning? Something like the following:mypy -p linkedin --namespace-packageswould be morally equivalent to:Then you'd just iterate over the elements?
Alternatively, if a
srcdirectory is given, just continue to recursively traverse until you find a concrete package and.pyfiles to start scanning.I'm sure both of these have subtitles that make it tricky. If we must, we'll have to auto-discover the subdirectories ourselves and pass them along on the command line.
ilevkivskyi commentedon Jun 16, 2019
Btw, it looks like this is related to #6385
gvanrossum commentedon Jun 17, 2019
I would rather not have mypy "do the import" -- that could run code which might have undesirable side effects, or it could just fail mysteriously. (In general we're pretty adamant that mypy shouldn't ever run your code in order to check it.) Also mypy won't know that the argument refers to a namespace package until it's too late.
I think the approach you mentioned that "does work",
mypy src/linkedin --namespace-packages, doesn't actually really work, because it ends up thinking that the fullly-qualified name ofbar.pyisfoo.bar, whereas it should really belinkedin.foo.bar. This distinction becomes important when other files are being checked.I suppose the most surprising part is that
MYPYPATH=src mypy -p linkedin --namespace-packagedoesn't work. Possibly we could make this work with some tweaks to the implementation of the-pflag inmypy/main.py. (Although it would be slightly inconsistent, since mypy's package discovery still wouldn't work if you were to writemypy src/linkedin/foo/bar.py-- that would still think the full name isfoo.bar.)Would you be interested in giving this a try?
warsaw commentedon Jun 26, 2019
Yes! I think it would be fine if the second invocation form was a little inconsistent.
Here's how I'm thinking about it. Let's say I do this:
That makes sense to me; I want to check the
linkedinnamespace anchored underneathsrc. What bothers me is that I have to create a dummy__init__.pyifile to make this work, and that this doesn't work the same way:Is that a reasonable thing to do?
gvanrossum commentedon Jun 26, 2019
I think we all agree that that should work. I was hoping that you'd try to come up with a PR for mypy to make it work that way, since the whole mypy team is basically super distracted by other priorities at the moment...
warsaw commentedon Jun 26, 2019
Fair enough! I'll give it a go, although it likely won't be until after the week of July 4th.
warsaw commentedon Jul 17, 2019
See the above PR. I've done internal testing and it works well for us.