mypyc-related stubtest crash

Bug Report

In typeshed CI stubtest crashes when run against fpdf2 2.7.5 and the current typeshed stubs. It works fine with fpdf2 2.7.4. Please see python/typeshed#10533, or to be more specific the traceback at https://github.com/python/typeshed/actions/runs/5930009431/job/16078844153?pr=10533:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/tmp/tmpo2a_8672/lib/python3.10/site-packages/mypy/stubtest.py", line 1891, in <module>
    sys.exit(main())
  File "/tmp/tmpo2a_8672/lib/python3.10/site-packages/mypy/stubtest.py", line 1887, in main
    return test_stubs(parse_options(sys.argv[1:]))
  File "/tmp/tmpo2a_8672/lib/python3.10/site-packages/mypy/stubtest.py", line 1760, in test_stubs
    for error in test_module(module):
  File "/tmp/tmpo2a_8672/lib/python3.10/site-packages/mypy/stubtest.py", line 223, in test_module
    yield from verify(stub, runtime, [module_name])
  File "/tmp/tmpo2a_8672/lib/python3.10/site-packages/mypy/stubtest.py", line 401, in verify_mypyfile
    yield from verify(stub_entry, runtime_entry, object_path + [entry])
  File "/tmp/tmpo2a_8672/lib/python3.10/site-packages/mypy/stubtest.py", line 543, in verify_typeinfo
    yield from verify(stub_to_verify, runtime_attr, object_path + [entry])
  File "/tmp/tmpo2a_8672/lib/python3.10/site-packages/mypy/stubtest.py", line [100](https://github.com/python/typeshed/actions/runs/5930009431/job/16078844153?pr=10533#step:5:101)6, in verify_funcitem
    for message in _verify_signature(stub_sig, runtime_sig, function_name=stub.name):
  File "/tmp/tmpo2a_8672/lib/python3.10/site-packages/mypy/stubtest.py", line 858, in _verify_signature
    yield from _verify_arg_default_value(stub_arg, runtime_arg)
  File "/tmp/tmpo2a_8672/lib/python3.10/site-packages/mypy/stubtest.py", line 632, in _verify_arg_default_value
    runtime_type = get_mypy_type_of_runtime_value(runtime_arg.default)
  File "/tmp/tmpo2a_8672/lib/python3.10/site-packages/mypy/stubtest.py", line 1563, in get_mypy_type_of_runtime_value
    return mypy.types.LiteralType(value=value, fallback=fallback)
  File "mypy/types.py", line 2584, in __init__
TypeError: union[int, str, bool, float] object expected; got None

This error seems to be generated by mypyc. I can't reproduce it locally, possibly because mypyc is not used?

Your Environment

Mypy version used: 1.5.1
Mypy command-line flags:

/tmp/tmpo2a_8672/bin/pip install fpdf2[]==2.7.5 mypy==1.5.1
MYPYPATH=/home/runner/work/typeshed/typeshed/stubs/fpdf2:/home/runner/work/typeshed/typeshed/stubs/Pillow /tmp/tmpo2a_8672/bin/python -m mypy.stubtest --custom-typeshed-dir /home/runner/work/typeshed/typeshed fpdf --allowlist /home/runner/work/typeshed/typeshed/stubs/fpdf2/@tests/stubtest_allowlist.txt

Python version used: 3.10.12

I can reproduce this crash locally on Python 3.9 and Python 3.10, but not Python 3.11

@srittau

I can reproduce this crash locally on Python 3.9 and Python 3.10, but not Python 3.11

And, as @srittau says, the crash only occurs with a compiled version of mypy, which makes it tricky to debug!

Looks like the TypeError is raised when mypy is trying to validate the default value typeshed gives for one of the parameters in this __init__ method: https://github.com/py-pdf/fpdf2/blob/ff55d983a0b6d601a78628a45747234fbca59e26/fpdf/encryption.py#L127-L135

I confirmed locally that applying this diff to typeshed's stubs makes the crash go away:

--- a/stubs/fpdf2/fpdf/encryption.pyi
+++ b/stubs/fpdf2/fpdf/encryption.pyi
@@ -65,15 +65,6 @@ class StandardSecurityHandler:
     k: str
     u: str

-    def __init__(
-        self,
-        fpdf: FPDF,
-        owner_password: str,
-        user_password: str | None = None,
-        permission: Incomplete | None = None,
-        encryption_method: EncryptionMethod | None = None,
-        encrypt_metadata: bool = False,
-    ) -> None: ...
     def generate_passwords(self, file_id) -> None: ...
     def get_encryption_obj(self) -> EncryptionDictionary: ...
     def encrypt(self, text: str | bytes | bytearray, obj_id) -> bytes: ...

The issue is this call here:

mypy/mypy/stubtest.py

Line 1563 in 7141d6b

return mypy.types.LiteralType(value=value, fallback=fallback)

When attempting to validate the permission parameter on fpdf.encryption.StandardSecurityHandler.__init__, stubtest ends up trying to call mypy.types.LiteralType(value=None, fallback=fpdf.enums.AccessPermission). That goes against the type annotation for mypy.types.LiteralType.__init__, which says that value should only ever be int, str, bool or float:

mypy/mypy/types.py

Lines 2560 to 2578 in 7141d6b

    
           class LiteralType(ProperType): 
        
               """The type of a Literal instance. Literal[Value] 
        
               A Literal always consists of: 
        
               1. A native Python object corresponding to the contained inner value 
        
               2. A fallback for this Literal. The fallback also corresponds to the 
        
                  parent type this Literal subtypes. 
        
               For example, 'Literal[42]' is represented as 
        
               'LiteralType(value=42, fallback=instance_of_int)' 
        
               As another example, `Literal[Color.RED]` (where Color is an enum) is 
        
               represented as `LiteralType(value="RED", fallback=instance_of_color)'. 
        
               """ 
        
               __slots__ = ("value", "fallback", "_hash") 
        
               def __init__(

Inserting a reveal_type() call indicates that mypy believes that value can only be an int or a str here, which is why the type violation is only being caught by the runtime checks that mypyc inserts when compiling mypy:

--- a/mypy/stubtest.py
+++ b/mypy/stubtest.py
@@ -1560,6 +1560,7 @@ def get_mypy_type_of_runtime_value(runtime: Any) -> mypy.types.Type | None:
     else:
         return fallback

+    reveal_type(value)
     return mypy.types.LiteralType(value=value, fallback=fallback)

> python runtests.py self
run self: ['C:\\Users\\alexw\\coding\\mypy\\venv\\Scripts\\python.exe', '-m', 'mypy', '--config-file', 'mypy_self_check.ini', '-p', 'mypy', '-p', 'mypyc']
mypy\stubtest.py:1563: note: Revealed type is "Union[builtins.int, builtins.str]"

Here is the reason why stubtest crashes on Python <=3.10 but not on Python 3.11. On Python 3.10:

Python 3.10.12 | packaged by Anaconda, Inc. | (main, Jul  5 2023, 19:01:18) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import fpdf.encryption
>>> fpdf.encryption.AccessPermission.all().name is None
True

On Python 3.11:

Python 3.11.4 | packaged by Anaconda, Inc. | (main, Jul  5 2023, 13:47:18) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import fpdf.encryption
>>> fpdf.encryption.AccessPermission.all().name
'PRINT_LOW_RES|MODIFY|COPY|ANNOTATION|FILL_FORMS|COPY_FOR_ACCESSIBILITY|ASSEMBLE|PRINT_HIGH_RES'

isinstance(fpdf.encryption.AccessPermission.all(), enum.Enum) evaluates to True, which means that value is set to fpdf.encryption.AccessPermission.all().name here:

mypy/mypy/stubtest.py

Lines 1556 to 1557 in 7141d6b

    
           elif isinstance(runtime, enum.Enum): 
        
               value = runtime.name

But on Python <=3.10, fpdf.encryption.AccessPermission.all().name is None, which triggers the crash.

The fpdf.encryption.AccessPermissions IntFlag is defined here: https://github.com/py-pdf/fpdf2/blob/34ff166de50a757c8cb32d6f047f0901e052d814/fpdf/enums.py#L801-L834

The reason why mypy doesn't pick up that runtime.name could be None here during the self-check is because enum.Flag violates the Liskov Substitution Principle. The name property can only ever be a str on an instance of an enum.Enum enumeration... unless that enumeration is a subclass of IntFlag, in which case it could also be None: https://github.com/python/typeshed/blob/32b750b6aa2396cdaa757da16da8cd8fb627c09f/stdlib/enum.pyi#L216-L217

I proposed a fix in #15933. The fix is a one-line change, though finding a way to reproduce the crash in the context of stubtest's test suite was a slight challenge 😄

Fix stubtest mypy enum.Flag edge case (#15933)

Navigation Menu

Repository navigation

mypyc-related stubtest crash #15923

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	class LiteralType(ProperType):
	"""The type of a Literal instance. Literal[Value]

	A Literal always consists of:

	1. A native Python object corresponding to the contained inner value
	2. A fallback for this Literal. The fallback also corresponds to the
	parent type this Literal subtypes.

	For example, 'Literal[42]' is represented as
	'LiteralType(value=42, fallback=instance_of_int)'

	As another example, `Literal[Color.RED]` (where Color is an enum) is
	represented as `LiteralType(value="RED", fallback=instance_of_color)'.
	"""

	__slots__ = ("value", "fallback", "_hash")

	def __init__(

mypyc-related stubtest crash #15923

Description

Activity

AlexWaygood commented on Aug 22, 2023

AlexWaygood commented on Aug 22, 2023

AlexWaygood commented on Aug 22, 2023

AlexWaygood commented on Aug 22, 2023

AlexWaygood commented on Aug 22, 2023

AlexWaygood commented on Aug 22, 2023

AlexWaygood commented on Aug 22, 2023

AlexWaygood commented on Aug 22, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions