Skip to content

Improve CLI, refactor and document stubgen - #6256

Merged
ilevkivskyi merged 35 commits into
python:masterfrom
ilevkivskyi:unify-stubgen
Jan 30, 2019
Merged

ilevkivskyi merged 35 commits into
python:masterfrom
ilevkivskyi:unify-stubgen

Conversation

@ilevkivskyi

Copy link
Copy Markdown
Member

This PR does several "infrastructure" changes to stubgen tool:

  • Update CLI and source discovery/collection to match the mypy CLI
  • Refactor the logic in stubgen.main() into independent functions
  • Use semantically analyzed Python ASTs
  • Separate inference of signatures from docs to a separate module stubdoc.py
  • Move some functions from stubgen.py to stubutil.py and reorganize the latter
  • Clean-up the test runner to make more use of DataSuite
  • Add documentation for the stubgen script

This also does few smaller things:

  • Minimize number of repeated hard-coded constants
  • Removed duplicate TODOs
  • Added dosctrings to stubgenc.py and (new) stubdoc.py
  • Avoid mutable defaults

This is not a pure refactoring, turning the semantic analysis on required some (although relatively small) changes in logic (because the sources should be semantically analyzed as a whole). It also required couple minor changes in semanal.py and build.py.

@ilevkivskyi
ilevkivskyi requested a review from JukkaL January 25, 2019 12:38
@ilevkivskyi
ilevkivskyi requested a review from msullivan January 25, 2019 13:37

@JukkaL JukkaL left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This really makes stubgen a much more useful tool! Additional thanks for cleaning up the code and writing docstrings.

This the first part of my review. I'll continue next week.

Some general things:

  • Some generated aliases may define a name with an underscore prefix (e.g. if I generate a stub for os). Should these be filtered out?
  • Similar to above, some aliases use names with an underscore prefix (e.g. if I generate stub for os). Should these be declared as x: Any instead?
  • Some generated constants are declared as x: Final. Should these be x: Final[Any] instead?

Comment thread mypy/stubgenc.py Outdated
Comment thread mypy/stubgenc.py
Comment thread mypy/stubgenc.py Outdated
Comment thread mypy/stubutil.py Outdated
Comment thread mypy/stubutil.py Outdated
Comment thread mypy/test/teststubgen.py Outdated
Comment thread mypy/test/teststubgen.py Outdated
Comment thread mypy/test/teststubgen.py Outdated
Comment thread mypy/test/teststubgen.py Outdated
@ilevkivskyi

Copy link
Copy Markdown
Member Author

@JukkaL Thanks for review! I implemented your comments. Regarding private names: it is now controlled by --include-private (previously this flag was only used for methods/functions). Regarding Final -- good catch, Final without argument is indeed invalid in stubs. (I added tests for both private names and final names).

A general comment, I didn't strive for completeness in this PR, small fixes (like above) can be added to this PR, but for all larger things I would rather open follow-up issues.

@ilevkivskyi

Copy link
Copy Markdown
Member Author

(AppVeyor failure is a flake)

@JukkaL JukkaL left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updates! This concludes my review; left a bunch of mostly minor nits. Feel free to ignore things that aren't relevant and create follow-up issues for things that are non-trivial to address.

Comment thread mypy/newsemanal/semanal.py
Comment thread mypy/semanal.py
Comment thread mypy/stubdoc.py Outdated
Comment thread mypy/stubdoc.py Outdated
Comment thread mypy/stubdoc.py Outdated
Comment thread mypy/stubgen.py Outdated
Comment thread mypy/stubgen.py
Comment thread mypy/stubgen.py Outdated
Comment thread mypy/stubgen.py Outdated
Comment thread mypy/stubgen.py Outdated
@ilevkivskyi

Copy link
Copy Markdown
Member Author

OK, playing with real stub generation discovered an issue: we should not add -> None: to functions without return statements if it is an abstract method.

Comment thread docs/source/stubgen.rst Outdated
Don't try to import modules, instead use mypy's normal mechanisms to find
sources. This will not find any C extension modules. Stubgen also uses
runtime introspection to find actual value of ``__all__``, so with this flag
the set of re-expoted names may be incomplete. This flag will be useful if

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

expoted

Comment thread docs/source/stubgen.rst Outdated

``--parse-only``
Don't perform mypy semantic analysis of source files. This may generate
worse stubs, in particular some module, class, and function aliases may

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

first comma should be a colon

@JukkaL

JukkaL commented Jan 29, 2019

Copy link
Copy Markdown
Collaborator

The changes in response to my review look good.

Comment thread test-data/unit/stubgen.test Outdated

class A(metaclass=ABCMeta):
@abstractmethod
def meth(self) -> None: ...

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the return type be Any?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The return is Any only if we run semantic analyzer (test name should end in _semanal), see a test below. But on the second thought it looks like it is not always necessary, in most common cases (like when we add the decorator here) it is better to return Any. I will fix this now.

Comment thread test-data/unit/stubgen.test Outdated

class A(metaclass=abc.ABCMeta):
@abc.abstractmethod
def meth(self) -> None: ...

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, should the return type be Any?

@ilevkivskyi

Copy link
Copy Markdown
Member Author

Oh, GitHub somehow completely screwed up my merge. I will try to fix it.

@ilevkivskyi

Copy link
Copy Markdown
Member Author

OK, it was actually my fault, I didn't add one file to the commit :P

@ilevkivskyi

Copy link
Copy Markdown
Member Author

Unless there are some objections, I am going to merge this soon after tests pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants