From 82ac1bbabc455ce91c8e9e6dfd728f1ce909c520 Mon Sep 17 00:00:00 2001 From: Jukka Lehtosalo Date: Fri, 27 Apr 2018 13:12:02 +0100 Subject: [PATCH 1/5] Document mypy daemon and the dmypy utility Also discuss how to use remote caching with the mypy daemon. --- docs/source/additional_features.rst | 36 +++++++++--- docs/source/index.rst | 1 + docs/source/mypy_daemon.rst | 88 +++++++++++++++++++++++++++++ 3 files changed, 118 insertions(+), 7 deletions(-) create mode 100644 docs/source/mypy_daemon.rst diff --git a/docs/source/additional_features.rst b/docs/source/additional_features.rst index 4ac46132e829..68d746a41cd9 100644 --- a/docs/source/additional_features.rst +++ b/docs/source/additional_features.rst @@ -1,13 +1,7 @@ Additional features ------------------- -Several mypy features are not currently covered by this tutorial, -including the following: - -- inheritance between generic classes -- compatibility and subtyping of generic types, including covariance of generic types -- ``super()`` - +This section discusses various features outside core mypy features. .. _attrs_package: @@ -177,6 +171,29 @@ commit id of the merge base produced by the git command above. The script will decompress the data so that mypy will start with a fresh ``.mypy_cache``. Finally, the script runs mypy normally. And that's all! +Caching with mypy daemon +======================== + +You can use the :ref:`mypy daemon ` for faster +incremental mypy runs, and it also supports remote caching. The remote +cache will significantly speed up the the first ``dmypy check`` run +after starting or restarting the daemon. + +The mypy daemon requires extra fine-grained dependency data in +the cache files which aren't included by default. To use caching with +the mypy daemon, use the ``--cache-fine-grained`` option in your CI +build:: + + $ mypy --cache-fine-grained + +You will also need to use the ``--use-fine-grained-cache`` option +with ``dymypy start`` or ``dmypy restart``. Example:: + + $ dmypy start -- --use-fine-grained-cache + +Now your first ``dmypy check`` run should be much faster, as it can use +cache information to avoid processing the whole program. + Refinements =========== @@ -187,6 +204,11 @@ at least if your codebase is hundreds of thousands of lines or more: from a previous run, there's no need to download the cache data and it's better to instead reuse the existing local cache data. +* If you use the mypy daemon, you may want to restart the daemon each time + after the merge base has changed to avoid processing a potentially large + number of changes in an incremental build, as this can be much slower + than downloading cache data and restarting the daemon. + * If the current local branch is based on a very recent master commit, the remote cache data may not yet be available for that commit, as there will necessarily be some latency to build the cache files. It diff --git a/docs/source/index.rst b/docs/source/index.rst index 582c1c4ee1b8..3556e23d34c3 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -29,6 +29,7 @@ Mypy is a static type checker for Python. additional_features command_line config_file + mypy_daemon python36 installed_packages faq diff --git a/docs/source/mypy_daemon.rst b/docs/source/mypy_daemon.rst new file mode 100644 index 000000000000..a0b9ad1530dc --- /dev/null +++ b/docs/source/mypy_daemon.rst @@ -0,0 +1,88 @@ +.. _mypy_daemon: + +Mypy daemon (mypy server) +========================= + +Instead of running mypy as a command-line tool, you can also run it as +a long-running daemon (server) process and use a command-line client to +control the server. This way mypy can perform type checking much faster, +since program state cached from previous runs is kept in memory and doesn't +have to be read from the file system on each run. The server also uses +finer-grained dependency tracking to reduce the amount of work that needs +to be done. + +If you have a large codebase to check, running mypy using the mypy +daemon can be *10 or more times faster* than the regular command-line +``mypy`` tool, especially if your workflow involves running mypy +repeatedly after small edits -- which is often a good idea, as this way +you'll find errors sooner. + +.. note:: + + The mypy daemon is experimental. In particular, the command-line + interface may change in future mypy releases. + +.. note:: + + The mypy daemon currently supports macOS and Linux only. + +Basic usage +*********** + +The client utility ``dmypy`` is used to control the mypy daemon. +Use ``dmypy start -- `` to start the daemon. You can use almost +arbitrary mypy flags after ``--``. The daemon will always run on the +current host. Example:: + + dmypy start -- --follow-imports=skip + +.. note:: + You'll need to use either the ``--follow-imports=skip`` or the + ``--follow-imports=error`` option with dmypy. + See :ref:`follow-imports` for details on how these work. + +The daemon will not type check anything after the start command. +Use ``dmypy check `` to check some files (or directories):: + + dmypy check prog.py pkg1/ pkg2/ + +You need to provide all files or directories you want to type check +(other than stubs) as arguments. + +The initial run will process all the code and may take a while to +finish, but subsequent runs will be quick, especially if you've only +changed a few files. + +Additional features +******************* + +You have precise control over the lifetime of the daemon process: + +* ``dymypy stop`` stops the daemon. + +* ``dmypy restart -- `` restarts the daemon. The flags are the same + as with ``dmypy start``. + +* Use ``dmypy start --timeout SECONDS -- `` (or + ``dmypy restart --timeout SECONDS -- ``) to automatically + shut down the daemon after inactivity. + +Use ``dmypy --help`` for help on additional commands and command-line +options not discussed here, and ``dmypy --help`` for help on +command-specific options. + +You can use a :ref:`remote cache ` to speed up the +initial ``dmypy check`` run. The speedup can be significant if +you have a large codebase. + +Limitations +*********** + +* Changes related to protocol classes are not reliably propagated. + +* You have to use either the ``--follow-imports=skip`` or + the ``--follow-imports=error`` option. This can be defined + through the command line or through a + :ref:`configuration file `. + +* Windows is not supported. From a59bac332121da659ad26824c7767a6da908889d Mon Sep 17 00:00:00 2001 From: Jukka Lehtosalo Date: Mon, 30 Apr 2018 17:41:16 +0100 Subject: [PATCH 2/5] Update based on feedback --- docs/source/additional_features.rst | 10 +++++--- docs/source/mypy_daemon.rst | 38 +++++++++++++++++++---------- 2 files changed, 31 insertions(+), 17 deletions(-) diff --git a/docs/source/additional_features.rst b/docs/source/additional_features.rst index 68d746a41cd9..4089bd5cbbfa 100644 --- a/docs/source/additional_features.rst +++ b/docs/source/additional_features.rst @@ -174,8 +174,8 @@ script will decompress the data so that mypy will start with a fresh Caching with mypy daemon ======================== -You can use the :ref:`mypy daemon ` for faster -incremental mypy runs, and it also supports remote caching. The remote +You can also use remote caching with the :ref:`mypy daemon `. +The remote cache will significantly speed up the the first ``dmypy check`` run after starting or restarting the daemon. @@ -186,8 +186,10 @@ build:: $ mypy --cache-fine-grained -You will also need to use the ``--use-fine-grained-cache`` option -with ``dymypy start`` or ``dmypy restart``. Example:: +This flag adds extra information for the daemon to the cache. In +order to use this extra information, you will also need to use the +``--use-fine-grained-cache`` option with ``dymypy start`` or +``dmypy restart``. Example:: $ dmypy start -- --use-fine-grained-cache diff --git a/docs/source/mypy_daemon.rst b/docs/source/mypy_daemon.rst index a0b9ad1530dc..fcdc1701795d 100644 --- a/docs/source/mypy_daemon.rst +++ b/docs/source/mypy_daemon.rst @@ -5,8 +5,9 @@ Mypy daemon (mypy server) Instead of running mypy as a command-line tool, you can also run it as a long-running daemon (server) process and use a command-line client to -control the server. This way mypy can perform type checking much faster, -since program state cached from previous runs is kept in memory and doesn't +send type-checking requests to the server. This way mypy can perform type +checking much faster, since program state cached from previous runs is kept +in memory and doesn't have to be read from the file system on each run. The server also uses finer-grained dependency tracking to reduce the amount of work that needs to be done. @@ -26,6 +27,12 @@ you'll find errors sooner. The mypy daemon currently supports macOS and Linux only. +.. note:: + + Each mypy daemon process supports one user and one set of source files, + and it can only process one type checking request at a time. You can + run multiple mypy daemon processes to type check multiple repositories. + Basic usage *********** @@ -38,20 +45,26 @@ current host. Example:: .. note:: You'll need to use either the ``--follow-imports=skip`` or the - ``--follow-imports=error`` option with dmypy. + ``--follow-imports=error`` option with dmypy because the current + implementation can't follow imports. See :ref:`follow-imports` for details on how these work. + You can also define these using a + :ref:`configuration file `. -The daemon will not type check anything after the start command. +The daemon will not type check anything when it's started. Use ``dmypy check `` to check some files (or directories):: dmypy check prog.py pkg1/ pkg2/ You need to provide all files or directories you want to type check -(other than stubs) as arguments. +(other than stubs) as arguments. This is a result of the +``--follow-imports`` restriction mentioned above. The initial run will process all the code and may take a while to finish, but subsequent runs will be quick, especially if you've only -changed a few files. +changed a few files. You can use :ref:`remote caching ` +to speed up the initial run. The speedup can be significant if +you have a large codebase. Additional features ******************* @@ -61,27 +74,26 @@ You have precise control over the lifetime of the daemon process: * ``dymypy stop`` stops the daemon. * ``dmypy restart -- `` restarts the daemon. The flags are the same - as with ``dmypy start``. + as with ``dmypy start``. This is equivalent to a stop command followed + by a start. * Use ``dmypy start --timeout SECONDS -- `` (or ``dmypy restart --timeout SECONDS -- ``) to automatically - shut down the daemon after inactivity. + shut down the daemon after inactivity. By default, the daemon runs + until it's explicitly stopped. Use ``dmypy --help`` for help on additional commands and command-line options not discussed here, and ``dmypy --help`` for help on command-specific options. -You can use a :ref:`remote cache ` to speed up the -initial ``dmypy check`` run. The speedup can be significant if -you have a large codebase. - Limitations *********** * Changes related to protocol classes are not reliably propagated. * You have to use either the ``--follow-imports=skip`` or - the ``--follow-imports=error`` option. This can be defined + the ``--follow-imports=error`` option because of an implementation + limitation. This can be defined through the command line or through a :ref:`configuration file `. From 606c26266983f52fd3183b769f4068dc3c6a40df Mon Sep 17 00:00:00 2001 From: Jukka Lehtosalo Date: Mon, 30 Apr 2018 19:10:14 +0100 Subject: [PATCH 3/5] Minor update --- docs/source/additional_features.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/additional_features.rst b/docs/source/additional_features.rst index 4089bd5cbbfa..669b733b7eb8 100644 --- a/docs/source/additional_features.rst +++ b/docs/source/additional_features.rst @@ -207,9 +207,9 @@ at least if your codebase is hundreds of thousands of lines or more: it's better to instead reuse the existing local cache data. * If you use the mypy daemon, you may want to restart the daemon each time - after the merge base has changed to avoid processing a potentially large - number of changes in an incremental build, as this can be much slower - than downloading cache data and restarting the daemon. + after the merge base or local branch has changed to avoid processing a + potentially large number of changes in an incremental build, as this can + be much slower than downloading cache data and restarting the daemon. * If the current local branch is based on a very recent master commit, the remote cache data may not yet be available for that commit, as From 2cef0ece2d8fb63be2f6f203836e28d1e4a49e39 Mon Sep 17 00:00:00 2001 From: Guido van Rossum Date: Mon, 30 Apr 2018 12:00:42 -0700 Subject: [PATCH 4/5] Reflow a paragraph. --- docs/source/additional_features.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/source/additional_features.rst b/docs/source/additional_features.rst index 669b733b7eb8..13f9b4cdd160 100644 --- a/docs/source/additional_features.rst +++ b/docs/source/additional_features.rst @@ -175,9 +175,8 @@ Caching with mypy daemon ======================== You can also use remote caching with the :ref:`mypy daemon `. -The remote -cache will significantly speed up the the first ``dmypy check`` run -after starting or restarting the daemon. +The remote cache will significantly speed up the the first ``dmypy check`` +run after starting or restarting the daemon. The mypy daemon requires extra fine-grained dependency data in the cache files which aren't included by default. To use caching with From c4a2bb7d2026c087db1119cfb5a2a145021e562e Mon Sep 17 00:00:00 2001 From: Guido van Rossum Date: Mon, 30 Apr 2018 12:01:48 -0700 Subject: [PATCH 5/5] Reflow another paragraph. --- docs/source/mypy_daemon.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/source/mypy_daemon.rst b/docs/source/mypy_daemon.rst index fcdc1701795d..b315804b1ed0 100644 --- a/docs/source/mypy_daemon.rst +++ b/docs/source/mypy_daemon.rst @@ -7,10 +7,9 @@ Instead of running mypy as a command-line tool, you can also run it as a long-running daemon (server) process and use a command-line client to send type-checking requests to the server. This way mypy can perform type checking much faster, since program state cached from previous runs is kept -in memory and doesn't -have to be read from the file system on each run. The server also uses -finer-grained dependency tracking to reduce the amount of work that needs -to be done. +in memory and doesn't have to be read from the file system on each run. +The server also uses finer-grained dependency tracking to reduce the amount +of work that needs to be done. If you have a large codebase to check, running mypy using the mypy daemon can be *10 or more times faster* than the regular command-line