From 20fe8eb14af0977c93d1999d4fe02021b6a9e58b Mon Sep 17 00:00:00 2001 From: Michael Lee Date: Sat, 16 Mar 2019 17:04:05 -0700 Subject: [PATCH 1/7] Add info about classes and types to 'getting started' docs While helping somebody use mypy, it came to my attention that our docs never actually quite explain how classes and types interact. This PR adds a new section named "Types and classes" to the "Getting started" docs that tries to fix this. This section focuses on introducing two ideas: 1. That you can use any Python class as a type 2. That if a variable has type T, it can also accept any instances of *subclasses* of type T. The examples also give us an opportunity to sneak in a brief mini-lesson on how to add type hints to classes, and what sorts of things do and do not get inferred. I also rewrote the next section about stubs/typeshed partly so it would dovetail better with this new section but also partly because I always felt that section was a little too dense and rushed. Three other minor changes made: - I added an example about sqlalchemy and sqlalchemy-stubs to one of the other pages. - I removed a line directing people to the mypy issue tracker if they have questions. - I dropped the term "library stubs" in favor of just "stub files". --- docs/source/class_basics.rst | 2 + docs/source/getting_started.rst | 155 +++++++++++++++++++++++++++----- docs/source/running_mypy.rst | 4 +- 3 files changed, 137 insertions(+), 24 deletions(-) diff --git a/docs/source/class_basics.rst b/docs/source/class_basics.rst index a89a7e2637e7e..37b1ca4afcd80 100644 --- a/docs/source/class_basics.rst +++ b/docs/source/class_basics.rst @@ -1,3 +1,5 @@ +.. _class-basics: + Class basics ============ diff --git a/docs/source/getting_started.rst b/docs/source/getting_started.rst index 017462d42c42d..8f27524c71c7a 100644 --- a/docs/source/getting_started.rst +++ b/docs/source/getting_started.rst @@ -4,7 +4,7 @@ Getting started =============== This chapter introduces some core concepts of mypy, including function -annotations, the ``typing`` module, library stubs, and more. +annotations, the ``typing`` module, stub files, and more. Be sure to read this chapter carefully, as the rest of the documentation may not make much sense otherwise. @@ -286,7 +286,7 @@ for example, when assigning an empty dictionary to some global value: You can teach mypy what type ``my_global_dict`` is meant to have by giving it a type hint. For example, if you knew this variable is supposed to be a dict of ints to floats, you could annotate it using either variable annotations -(introduced in Python 3.6 by :ref:`PEP 526 `_) or using a comment-based +(introduced in Python 3.6 by `PEP 526 `_) or using a comment-based syntax like so: .. code-block:: python @@ -300,34 +300,144 @@ syntax like so: .. _pep526: https://www.python.org/dev/peps/pep-0526/ -.. _stubs-intro: +Types and classes +***************** + +So far, we've only seen examples of pre-existing types like the ``int`` +or ``float`` builtins, or the ``List`` and ``Dict`` types from the typing +module. However, these aren't the only types you can use: in fact, you can +use any Python class as a type! + +For example, suppose you've defined a custom class representing a bank account: + +.. code-block:: python + + class BankAccount: + # Note: It is ok to omit type hints for the "self" parameter. + # Mypy will infer the correct type. + + def __init__(self, account_name: str, initial_balance: int = 0) -> None: + # Note: Mypy will infer the correct types of your fields + # based on the types of the parameters. + self.account_name = account_name + self.balance = initial_balance + + def deposit(self, amount: int) -> None: + self.balance += amount + + def withdraw(self, amount: int) -> None: + self.balance -= amount + + def overdrawn(self) -> bool: + return self.balance < 0 + +You can declare that a function will accept any instance of your class +by simply annotating the parameters with ``BankAccount``: + +.. code-block:: python + + def transfer(src: BankAccount, dst: BankAccount, amount: int) -> None: + src.withdraw(amount) + dst.deposit(amount) + + account_1 = BankAccount('Alice', 400) + account_2 = BankAccount('Bob', 200) + transfer(account_1, account_2, 50) + +In fact, the ``transfer`` function we wrote above can accept more then just +instances of ``BankAccount``: it can also accept any instance of a *subclass* +of ``BankAccount``. For example, suppose you write a new class that looks like this: + +.. code-block:: python + + class AuditedBankAccount(BankAccount): + def __init__(self, account_name: str, initial_balance: int = 0) -> None: + super().__init__(account_name, initial_balance) + + # In this case, mypy can't infer the exact type of this field + # based on the information available in this constructor, so we + # need to add a type annotation. + self.audit_log: List[str] = [] -Library stubs and typeshed -************************** + def deposit(self, amount: int) -> None: + self.audit_log.append(f"Deposited {amount}") + self.balance += amount -Mypy uses library *stubs* to type check code interacting with library -modules, including the Python standard library. A library stub defines -a skeleton of the public interface of the library, including classes, -variables and functions, and their types. Mypy ships with stubs from -the `typeshed `_ project, which -contains library stubs for the Python builtins, the standard library, -and selected third-party packages. + def withdraw(self, amount: int) -> None: + self.audit_log.append(f"Withdrew {amount}") + self.balance -= amount -For example, consider this code: +Since ``AuditedBankAccount`` is a subclass of ``BankAccount``, we can directly pass +in instances of it into our ``transfer`` function: .. code-block:: python - x = chr(4) + audited = AuditedBankAccount('Charlie', 300) + transfer(account_1, audited, 100) # Type checks! + +This behavior is actually a fundamental aspect of the PEP 484 type system: when +we annotate some variable with a type ``T``, we are actually telling mypy that +variable can be assigned an instance of ``T``, or an instance of a *subclass* of ``T``. +The same rule applies to type hints on parameters or fields. + +See :ref:`class-basics` to learn more about how to work with code involving classes. + + +.. _stubs-intro: + +Stubs files and typeshed +************************ + +Mypy also understands how to work with classes found in the standard library. +For example, here is a function which uses the ``Path`` object from the +`pathlib standard library module `_: + +.. code-block:: python + + from pathlib import Path + + def load_template(template_path: Path, name: str) -> str: + # Mypy understands that 'file_path.read_text()' returns a str... + template = template_path.read_text() + + # ...so understands this line type checks. + return template.replace('USERNAME', name) + +This behavior may surprise you if you're familiar with how +Python internally works. The standard library does not use type hints +anywhere, so how did mypy know that ``Path.read_text()`` returns a ``str``, +or that ``str.replace(...)`` accepts exactly two ``str`` arguments? + +The answer is that mypy comes bundled with *stub files* from the +the `typeshed `_ project. + +A *stub file* is a file containing a skeleton of the public interface +of that Python module, including classes, variables, functions -- and +most importantly, their types. Typeshed is a collection of stub files +for modules in the standard library and select third party libraries +and is what mypy used to identify the correct types for our example above. + +Mypy can understand third party libraries in the same way as long as those +libraries come bundled with stub files. (See :ref:`installed-packages` +for more details on how this is done). However, if the third party library +does *not* come bundled with type hints, mypy will not try and guess what +the types are: it'll assume the entire library is +:ref:`dynamically typed ` and report an error +whenever you import the library. + +We discuss strategies for handling this category of errors in +:ref:`ignore-missing-imports`, but in short, there are three things you can do: + +1. Look for a separate package containing stubs for that library. For example, + if you want type hints for the `sqlalchemy `_ + library, you can install + `sqlalchemy-stubs `_. + +2. :ref:`Create your own stub files ` for that library. -Without a library stub, mypy would have no way of inferring the type of ``x`` -and checking that the argument to ``chr`` has a valid type. +3. :ref:`Silence the errors ` and accept that + any uses of the library will remain dynamically typed. -Mypy complains if it can't find a stub (or a real module) for a -library module that you import. Some modules ship with stubs that mypy -can automatically find, or you can install a 3rd party module with -additional stubs (see :ref:`installed-packages` for details). You can -also :ref:`create stubs ` easily. We discuss ways of -silencing complaints about missing stubs in :ref:`ignore-missing-imports`. Configuring mypy **************** @@ -382,7 +492,6 @@ resources: you encounter problems. * You can ask questions about mypy in the - `mypy issue tracker `_ and typing `Gitter chat `_. You can also continue reading this document and skip sections that diff --git a/docs/source/running_mypy.rst b/docs/source/running_mypy.rst index 96f95b81e461a..424ae822e8753 100644 --- a/docs/source/running_mypy.rst +++ b/docs/source/running_mypy.rst @@ -169,7 +169,9 @@ are trying to use has done neither of these things. In that case, you can try: 1. Searching to see if there is a :ref:`PEP 561 compliant stub package `. corresponding to your third party library. Stub packages let you install - type hints independently from the library itself. + type hints independently from the library itself. For example, if you want + type hints for the `sqlalchemy `_ library, + you can install `sqlalchemy-stubs `_. 2. :ref:`Writing your own stub files ` containing type hints for the library. You can point mypy at your type hints either by passing From c3511a437f2c09f40367ec360c23f076ffbb293a Mon Sep 17 00:00:00 2001 From: Jingchen Ye <11172084+97littleleaf11@users.noreply.github.com> Date: Sun, 1 May 2022 11:21:33 +0800 Subject: [PATCH 2/7] Update docs Co-authored-by: AlexWaygood --- docs/source/getting_started.rst | 55 +++++------------------------- docs/source/installed_packages.rst | 9 +++++ docs/source/running_mypy.rst | 2 +- 3 files changed, 19 insertions(+), 47 deletions(-) diff --git a/docs/source/getting_started.rst b/docs/source/getting_started.rst index 52008ddd62995..9bb519aa8ea6e 100644 --- a/docs/source/getting_started.rst +++ b/docs/source/getting_started.rst @@ -327,9 +327,9 @@ Types and classes ***************** So far, we've only seen examples of pre-existing types like the ``int`` -or ``float`` builtins, or the ``List`` and ``Dict`` types from the typing -module. However, these aren't the only types you can use: in fact, you can -use any Python class as a type! +or ``float`` builtins, or generic types from ``collections.abc`` and +``typing``, such as ``Iterable``. However, these aren't the only types you can +use: in fact, you can use any Python class as a type! For example, suppose you've defined a custom class representing a bank account: @@ -442,49 +442,12 @@ and is what mypy used to identify the correct types for our example above. Mypy can understand third party libraries in the same way as long as those libraries come bundled with stub files or have inline annotations that mypy -can automatically find. (see :ref:`fix-missing-imports` and -:ref:`installed-packages` for the details). However, if the third party library -does *not* come bundled with type hints, mypy will not try and guess what -the types are: it'll assume the entire library is -:ref:`dynamically typed ` and report an error -whenever you import the library. - -We discuss strategies for handling this category of errors in -:ref:`ignore-missing-imports`, but in short, there are three things you can do: - -1. Look for a separate package containing stubs for that library. For example, - you can install the stubs for the ``requests`` package like this: - -.. code-block:: shell - - python3 -m pip install types-requests - -The stubs are usually packaged in a distribution named -``types-``. Note that the distribution name may be -different from the name of the package that you import. For example, -``types-PyYAML`` contains stubs for the ``yaml`` package. Mypy can -often suggest the name of the stub distribution: - -.. code-block:: text - - prog.py:1: error: Library stubs not installed for "yaml" (or incompatible with Python 3.8) - prog.py:1: note: Hint: "python3 -m pip install types-PyYAML" - ... - -.. note:: - - Starting in mypy 0.900, most third-party package stubs must be - installed explicitly. This decouples mypy and stub versioning, - allowing stubs to updated without updating mypy. This also allows - stubs not originally included with mypy to be installed. Earlier - mypy versions included a fixed set of stubs for third-party - packages. - -2. :ref:`Create your own stub files ` for that library. - -3. :ref:`Silence the errors ` and accept that - any uses of the library will remain dynamically typed. - +can automatically find. (see :ref:`installed-packages` for the details). +However, if the third party library does *not* come bundled with type hints, +mypy will not try and guess what the types are: it'll assume the entire library +is :ref:`dynamically typed ` and report an error whenever you +import the library. We discuss strategies for handling this category of errors +in :ref:`ignore-missing-imports`. Configuring mypy **************** diff --git a/docs/source/installed_packages.rst b/docs/source/installed_packages.rst index 036185c818e5c..396e616be1e7a 100644 --- a/docs/source/installed_packages.rst +++ b/docs/source/installed_packages.rst @@ -25,6 +25,15 @@ you can create such packages. :pep:`561` specifies how a package can declare that it supports type checking. +.. note:: + + Starting in mypy 0.900, most third-party package stubs must be + installed explicitly. This decouples mypy and stub versioning, + allowing stubs to updated without updating mypy. This also allows + stubs not originally included with mypy to be installed. Earlier + mypy versions included a fixed set of stubs for third-party + packages. + Using installed packages with mypy (PEP 561) ******************************************** diff --git a/docs/source/running_mypy.rst b/docs/source/running_mypy.rst index de78cc3af5a46..269e3b9b1570d 100644 --- a/docs/source/running_mypy.rst +++ b/docs/source/running_mypy.rst @@ -324,7 +324,7 @@ this error, try: In some rare cases, you may get the "Cannot find implementation or library stub for module" error even when the module is installed in your system. This can happen when the module is both missing type hints and is installed -on your system in a unconventional way. +on your system in an unconventional way. In this case, follow the steps above on how to handle :ref:`missing type hints in third party libraries `. From 36301864b5264fcb7b57f37f67660e98cbadf7e4 Mon Sep 17 00:00:00 2001 From: Jingchen Ye <11172084+97littleleaf11@users.noreply.github.com> Date: Sun, 1 May 2022 11:45:50 +0800 Subject: [PATCH 3/7] Fix --- docs/source/getting_started.rst | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/docs/source/getting_started.rst b/docs/source/getting_started.rst index 98d68cb5c0f74..688c8ef32e1da 100644 --- a/docs/source/getting_started.rst +++ b/docs/source/getting_started.rst @@ -317,11 +317,6 @@ syntax like so: # If you're using Python 3.6+ my_global_dict: Dict[int, float] = {} - # If you want compatibility with even older versions of Python - my_global_dict = {} # type: Dict[int, float] - -.. _pep526: https://www.python.org/dev/peps/pep-0526/ - .. _stubs-intro: @@ -400,7 +395,7 @@ in instances of it into our ``transfer`` function: audited = AuditedBankAccount('Charlie', 300) transfer(account_1, audited, 100) # Type checks! -This behavior is actually a fundamental aspect of the PEP 484 type system: when +This behavior is actually a fundamental aspect of the PEP 484 type system: when we annotate some variable with a type ``T``, we are actually telling mypy that variable can be assigned an instance of ``T``, or an instance of a *subclass* of ``T``. The same rule applies to type hints on parameters or fields. @@ -504,6 +499,7 @@ resources: you encounter problems. * You can ask questions about mypy in the + `mypy issue tracker `_ and typing `Gitter chat `_. You can also continue reading this document and skip sections that From 0f03e2b0c68d6a82adb3735c39cf9fce2cf59fc0 Mon Sep 17 00:00:00 2001 From: Jingchen Ye <11172084+97littleleaf11@users.noreply.github.com> Date: Sun, 1 May 2022 12:02:21 +0800 Subject: [PATCH 4/7] Add back the example introduced in PR9973 --- docs/source/getting_started.rst | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/docs/source/getting_started.rst b/docs/source/getting_started.rst index 688c8ef32e1da..2478934f21f55 100644 --- a/docs/source/getting_started.rst +++ b/docs/source/getting_started.rst @@ -443,8 +443,29 @@ can automatically find. (see :ref:`installed-packages` for the details). However, if the third party library does *not* come bundled with type hints, mypy will not try and guess what the types are: it'll assume the entire library is :ref:`dynamically typed ` and report an error whenever you -import the library. We discuss strategies for handling this category of errors -in :ref:`ignore-missing-imports`. +import the library. + +You can install the stubs for third-party packages like this: + +.. code-block:: shell + + $ python3 -m pip install types-requests + +The stubs are usually packaged in a distribution named +``types-``. Note that the distribution name may be +different from the name of the package that you import. For example, +``types-PyYAML`` contains stubs for the ``yaml`` package. Mypy can +often suggest the name of the stub distribution: + +.. code-block:: text + + prog.py:1: error: Library stubs not installed for "yaml" (or incompatible with Python 3.8) + prog.py:1: note: Hint: "python3 -m pip install types-PyYAML" + ... + +You can also :ref:`create +stubs ` easily. We discuss strategies for handling errors +about missing stubs in :ref:`ignore-missing-imports`. Configuring mypy **************** From 310cf34c61a9969fe102940bbcd9ad827bbc2fa4 Mon Sep 17 00:00:00 2001 From: Jingchen Ye <11172084+97littleleaf11@users.noreply.github.com> Date: Sun, 1 May 2022 12:06:09 +0800 Subject: [PATCH 5/7] Fix --- docs/source/getting_started.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/source/getting_started.rst b/docs/source/getting_started.rst index 2478934f21f55..a83b38c64a6ef 100644 --- a/docs/source/getting_started.rst +++ b/docs/source/getting_started.rst @@ -317,7 +317,6 @@ syntax like so: # If you're using Python 3.6+ my_global_dict: Dict[int, float] = {} - .. _stubs-intro: Types and classes @@ -459,9 +458,9 @@ often suggest the name of the stub distribution: .. code-block:: text - prog.py:1: error: Library stubs not installed for "yaml" (or incompatible with Python 3.8) - prog.py:1: note: Hint: "python3 -m pip install types-PyYAML" - ... + prog.py:1: error: Library stubs not installed for "yaml" (or incompatible with Python 3.8) + prog.py:1: note: Hint: "python3 -m pip install types-PyYAML" + ... You can also :ref:`create stubs ` easily. We discuss strategies for handling errors From ea58976efbdf5839e692b778ba9b4707b6f2ab83 Mon Sep 17 00:00:00 2001 From: Jingchen Ye <11172084+97littleleaf11@users.noreply.github.com> Date: Fri, 6 May 2022 09:46:38 +0800 Subject: [PATCH 6/7] Update --- docs/source/getting_started.rst | 32 ++++++++++++-------------------- 1 file changed, 12 insertions(+), 20 deletions(-) diff --git a/docs/source/getting_started.rst b/docs/source/getting_started.rst index a83b38c64a6ef..6d92ce30d7237 100644 --- a/docs/source/getting_started.rst +++ b/docs/source/getting_started.rst @@ -317,7 +317,6 @@ syntax like so: # If you're using Python 3.6+ my_global_dict: Dict[int, float] = {} -.. _stubs-intro: Types and classes ***************** @@ -372,11 +371,7 @@ of ``BankAccount``. For example, suppose you write a new class that looks like t class AuditedBankAccount(BankAccount): def __init__(self, account_name: str, initial_balance: int = 0) -> None: super().__init__(account_name, initial_balance) - - # In this case, mypy can't infer the exact type of this field - # based on the information available in this constructor, so we - # need to add a type annotation. - self.audit_log: List[str] = [] + self.audit_log: list[str] = [] def deposit(self, amount: int) -> None: self.audit_log.append(f"Deposited {amount}") @@ -428,23 +423,20 @@ anywhere, so how did mypy know that ``Path.read_text()`` returns a ``str``, or that ``str.replace(...)`` accepts exactly two ``str`` arguments? The answer is that mypy comes bundled with *stub files* from the -the `typeshed `_ project. +the `typeshed `_ project, which +contains stub files for the Python builtins, the standard library, +and selected third-party packages. A *stub file* is a file containing a skeleton of the public interface of that Python module, including classes, variables, functions -- and -most importantly, their types. Typeshed is a collection of stub files -for modules in the standard library and select third party libraries -and is what mypy used to identify the correct types for our example above. - -Mypy can understand third party libraries in the same way as long as those -libraries come bundled with stub files or have inline annotations that mypy -can automatically find. (see :ref:`installed-packages` for the details). -However, if the third party library does *not* come bundled with type hints, -mypy will not try and guess what the types are: it'll assume the entire library -is :ref:`dynamically typed ` and report an error whenever you -import the library. - -You can install the stubs for third-party packages like this: +most importantly, their types. + +Mypy complains if it can't find a stub (or a real module) for a +library module that you import. Some modules ship with stubs or inline +annotations that mypy can automatically find, or you can install +additional stubs using pip (see :ref:`fix-missing-imports` and +:ref:`installed-packages` for the details). For example, you can install +the stubs for the ``requests`` package like this: .. code-block:: shell From b75b855cc47f546a4201693ec67b37862f0936c4 Mon Sep 17 00:00:00 2001 From: Jingchen Ye <11172084+97littleleaf11@users.noreply.github.com> Date: Fri, 6 May 2022 09:49:57 +0800 Subject: [PATCH 7/7] Update --- docs/source/running_mypy.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/source/running_mypy.rst b/docs/source/running_mypy.rst index cdf0b480f901c..46ad2c65c386a 100644 --- a/docs/source/running_mypy.rst +++ b/docs/source/running_mypy.rst @@ -177,9 +177,7 @@ If you are getting this error, try: 2. Searching to see if there is a :ref:`PEP 561 compliant stub package `. corresponding to your third party library. Stub packages let you install - type hints independently from the library itself. For example, if you want - type hints for the `sqlalchemy `_ library, - you can install `sqlalchemy-stubs `_. + type hints independently from the library itself. For example, if you want type hints for the ``django`` library, you can install the `django-stubs `_ package.