What names should be included in stubs? #1287

I'm hoping to create or improve a tool that can automatically check whether stubs are comprehensive. But for such a tool to work reliably, we need to define what names should actually be in the stubs.

I'm assuming the approach would be something like:

• Find all the names defined in the module's __dict__ at runtime.
• Find all the names in the stub.
• Compare the two sets of names. If there is an inconsistency, use some set of heuristics to decide if it indicates a bug in the stub.

This is the approach taken by the stubtest.py tool that currently exists in the mypy repo. I've done some very preliminary work on a similar tool (http://github.com/JelleZijlstra/stubcheck).

For now, I want to focus only on finding names that are missing in the stub or at runtime. There's more we can do, like making sure type signatures are consistent, but that's for another day.

Here's a few rules we could consider:

• If there is an __all__ at runtime, all names in __all__ must be in the stub.
• All documented names must be in the stub. Implementing this would require parsing Sphinx documentation or something like that.
• If there is no __all__ and we can't use documentation:
  • Private names (starting with _) don't need to be in the stub.
  • When you're creating a stub for module a, names for which inspect.getmodule() either returns None at runtime or returns a different module than a, don't need to be in the stub. This rule means the stub doesn't have to include names imported from other modules. It also means that it doesn't have to include types that don't set a __module__ (like integer constants). In my experience, those are often intended to be private but not actually marked as private.
  • Names defined at runtime whose inspect.getmodule is the module being inspected should be included. This will include some functions that may have been intended to be private but aren't, but my hope is that there are few enough of those that it's not going to be a big problem.
• All names in the stub must also be defined at runtime, except for names that start with _ or that are imported without as.

And here are some complications we should be aware of:

• Some names only exist on some platforms. sys.platform checks in stubs aren't always sufficient to cover all such cases.
• Some names are imported from other modules but still part of the public API of a module (for example, os.scandir is actually defined in the posix module).
• Many modules use some kind of weird dynamism that may cause trouble.
• In the stub, we should make sure to parse all the version_info/platform correctly.

Let me know if there's some other heuristic we can use.