[mypyc] Pickle test failure with Python 3.14.0a1

After being a bit late with Python 3.13, I figured to start testing 3.14 early this time.

With Python 3.14.0a1 the following fails:

# native.py
from typing import Any

def dec(x: Any) -> Any:
    return x

@dec
class D:
    x: int

class E(D):
    y: int

# driver.py
from native import D, E

import pickle

assert not hasattr(D, '__mypyc_attrs__')
assert E.__mypyc_attrs__ == ('y', '__dict__')

e = E()
e.x = 10
e.y = 20

assert e.__getstate__() == {'y': 20, '__dict__': {'x': 10}}
e2 = pickle.loads(pickle.dumps(e))
assert e is not e2 and e.x == e2.x and e.y == e2.y

rm -rf build
rm *.so
python3.14 -m mypyc native.py
python3.14 -c 'import driver.py'

The error message

$ python -c "import driver.py"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import driver.py
  File ".../driver.py", line 14, in <module>
    assert e is not e2 and e.x == e2.x and e.y == e2.y
                                  ^^^^
AttributeError: 'E' object has no attribute 'x'

I bisected this to python/cpython#123192 upstream.

--
This is also part of the mypyc test suite

pytest -n0 mypyc/test/test_run.py::TestRun::run-classes.test::testPickling

I suspect it's something funky when calling PyObject_SetAttr(obj, "__dict__", {"x": 10}), which is what mypyc will do here when unpickling

With a debug Python build you also get a segfault following the AttributeError:

Assertion failed: (0), function _PyObject_InlineValuesConsistencyCheck, file dictobject.c, line 7470.
zsh: abort      python driver.py

In lldb:

  * frame #4: 0x000000010031b524 python`_PyObject_InlineValuesConsistencyCheck.cold.3 at dictobject.c:7470:5 [opt]
    frame #5: 0x00000001000d3e54 python`_PyObject_InlineValuesConsistencyCheck(obj=<unavailable>) at dictobject.c:7470:5 [opt]
    frame #6: 0x00000001000d3b94 python`_PyObject_SetManagedDict(obj=0x0000000100bc7420, new_dict=0x0000000000000000) at dictobject.c:7114:5 [opt]
    frame #7: 0x00000001000d3fb8 python`PyObject_ClearManagedDict(obj=<unavailable>) at dictobject.c:7176:9 [opt]
    frame #8: 0x0000000100c67088 native.cpython-314d-darwin.so`E_dealloc [inlined] E_clear at __native.c:50:5 [opt]
    frame #9: 0x0000000100c67024 native.cpython-314d-darwin.so`E_dealloc at __native.c:59:5 [opt]
    frame #10: 0x00000001000ebe0c python`_Py_Dealloc(op=0x0000000100bc7420) at object.c:2933:5 [opt]
    frame #11: 0x00000001000d41c0 python`dictkeys_decref [inlined] Py_DECREF(filename=<unavailable>, lineno=476, op=0x0000000100bc7420) at refcount.h:367:9 [opt]
    frame #12: 0x00000001000d4174 python`dictkeys_decref [inlined] Py_XDECREF(op=0x0000000100bc7420) at refcount.h:476:9 [opt]
    frame #13: 0x00000001000d4170 python`dictkeys_decref(interp=<unavailable>, dk=0x0000000100eda3b0, use_qsbr=false) at dictobject.c:460:17 [opt]
    frame #14: 0x00000001000cfc2c python`dict_dealloc(self=0x0000000100a6c7d0) at dictobject.c:0 [opt]
    frame #15: 0x00000001000ebe0c python`_Py_Dealloc(op=0x0000000100a6c7d0) at object.c:2933:5 [opt]
    ...

Still an issue with 3.14.0a2

Still an issue with 3.14.0a3 as well.
Edit: And with 3.14.0a4 too.

A possible workaround could be to disable the Py_TPFLAGS_INLINE_VALUES here:

mypy/mypyc/lib-rt/misc_ops.c

Lines 299 to 304 in f44a60d

    
           #if PY_MINOR_VERSION == 11 
        
               // This is a hack. Python 3.11 doesn't include good public APIs to work with managed 
        
               // dicts, which are the default for heap types. So we try to opt-out until Python 3.12. 
        
               t->ht_type.tp_flags &= ~Py_TPFLAGS_MANAGED_DICT; 
        
           #endif 
        
               return (PyObject *)t;

 #if PY_MINOR_VERSION == 11
     // This is a hack. Python 3.11 doesn't include good public APIs to work with managed
     // dicts, which are the default for heap types. So we try to opt-out until Python 3.12.
     t->ht_type.tp_flags &= ~Py_TPFLAGS_MANAGED_DICT;
+#elif PY_MINOR_VERSION == 14
+    t->ht_type.tp_flags &= ~Py_TPFLAGS_INLINE_VALUES;

With that the test case passes again (and no other test is failing because of it). However, I'm unsure that's the correct solution. With python/cpython#123192 Py_TPFLAGS_INLINE_VALUES is automatically set for type->tp_itemsize == 0 which is the case here.

@JukkaL

From what I can tell the testPickling case also had issues when adding support for 3.12.

/CC @JukkaL

This was fixed in python/cpython#134859 which will be included in Python 3.14.0b3.

Navigation Menu

Repository navigation

[mypyc] Pickle test failure with Python 3.14.0a1 #17973

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	#if PY_MINOR_VERSION == 11
	// This is a hack. Python 3.11 doesn't include good public APIs to work with managed
	// dicts, which are the default for heap types. So we try to opt-out until Python 3.12.
	t->ht_type.tp_flags &= ~Py_TPFLAGS_MANAGED_DICT;
	#endif
	return (PyObject *)t;

[mypyc] Pickle test failure with Python 3.14.0a1 #17973

Description

Activity

hauntsaninja commented on Oct 17, 2024

cdce8p commented on Nov 20, 2024

cdce8p commented on Dec 18, 2024

cdce8p commented on Feb 1, 2025

cdce8p commented on Feb 1, 2025

cdce8p commented on May 30, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions