Difference in behavior between pure Python pickle and C extension _pickle when unpickling instance of instance of metaclass

Bug report

When pickling and unpickling an instance of an instance of a metaclass, _pickle reaches into the class object directly and calls tp_new from the C side, whereas pickle uses cls.__new__, which triggers a custom __getattribute__ if defined. Whether this is a bug or just an intentional optimization I'm not sure but at the very least one there is an observable difference in behavior in that unpickling from a distribution where _pickle is available does not call __getattribute__ but unpickling from a distribution where _pickle is not available does call __getattribute__.

MWE

(Works on all available CPythons on Compiler Explorer, 3.5 - 3.11)
https://godbolt.org/z/x76bs8dzv

To simulate different distributions having _pickle available or not, we can use a meta path entry to cause it to fail to import, causing pickle to fall back to the pure Python version.

import importlib.abc
import sys

class HideModuleFinder(importlib.abc.MetaPathFinder):
    def __init__(self, hidden):
        self.hidden = set(hidden)

    def find_spec(self, fullname, path, target=None):
        if fullname in self.hidden:
            raise ImportError("Module is hidden")
        return None  # let next finder try

def install(hidden):
    sys.meta_path.insert(0, HideModuleFinder(hidden))

hide_pickle = False # change me for testing different behavior

if hide_pickle:
    install({"_pickle"})

import pickle # must be done after meta path hook

class Meta(type):
    def __getattribute__(self, item):
        print("__getattribute__ called with", item)
        return type.__getattribute__(self, item)

MyClass = Meta("MyClass", (), {})

obj = MyClass()
print("PICKLING")
obj_str = pickle.dumps(obj)

print("UNPICKLING")
new_obj = pickle.loads(obj_str)

Output with hide_pickle = True

PICKLING
__getattribute__ called with __reduce__
__getattribute__ called with __dict__
__getattribute__ called with __slots__
__getattribute__ called with __new__
__getattribute__ called with __qualname__
__getattribute__ called with __module__
UNPICKLING
__getattribute__ called with __new__

Output with hide_pickle = False

PICKLING
__getattribute__ called with __reduce__
__getattribute__ called with __dict__
__getattribute__ called with __slots__
__getattribute__ called with __qualname__
__getattribute__ called with __module__
UNPICKLING

Linked PRs

gh-105250: Fix NEWOBJ handling of custom metaclasses in C pickle #155920

@corona10

Hi @corona10 @hugovk, I'd like to work on this issue as part of the PyCon KR 2026 sprint.

I confirmed it still reproduces on current main (70fdc96).

One subtlety I noticed while investigating: the C unpickler is not always "metaclass-free". load_newobj in Modules/_pickle.c calls tp_new directly (even though the comment above the call says the intent is to call cls.__new__(cls, *args, **kwargs)), but when the class defines __new__ in Python, slot_tp_new itself looks __new__ up on the type, so the metaclass hook does fire even under _pickle. So whether the hook runs currently depends on whether the class being unpickled happens to define __new__ in Python.

Reproducer and output on main

from test.support.import_helper import import_fresh_module

c_pickle = import_fresh_module("pickle", fresh=["_pickle"])
py_pickle = import_fresh_module("pickle", blocked=["_pickle"])

LOG = []

class Meta(type):
    def __getattribute__(cls, name):
        LOG.append(name)
        return super().__getattribute__(name)

class InheritsNew(metaclass=Meta):   # tp_new is object.__new__ (a real C slot)
    def __init__(self, value):
        self.value = value

class DefinesNew(metaclass=Meta):    # tp_new is slot_tp_new
    def __new__(cls, *args):
        return super().__new__(cls)
    def __init__(self, value):
        self.value = value

def lookups(mod, obj, protocol):
    """Attribute lookups the metaclass observes while unpickling."""
    data = mod.dumps(obj, protocol=protocol)
    LOG.clear()
    mod.loads(data)
    return sorted(set(LOG))

for cls in (InheritsNew, DefinesNew):
    for protocol in (2, 5):
        c = lookups(c_pickle, cls(42), protocol)
        py = lookups(py_pickle, cls(42), protocol)
        print(f"{cls.__name__} / protocol {protocol}: C={c!s:<13} "
              f"py={py!s:<13} {'DIVERGES' if c != py else 'same'}")

InheritsNew / protocol 2: C=[]            py=['__new__']   DIVERGES
InheritsNew / protocol 5: C=[]            py=['__new__']   DIVERGES
DefinesNew / protocol 2: C=['__new__']   py=['__new__']   same
DefinesNew / protocol 5: C=['__new__']   py=['__new__']   same

I'm planning to keep the fix as minimal as possible (mirroring the pure-Python cls.__new__ lookup on the _pickle.c side). If you think this issue isn't a good fit for the sprint, please let me know and I'll pick another one.

@chaerrypick01

@chaerrypick01 I think that it's good to go

pythongh-105250: Fix NEWOBJ handling of custom metaclasses in C pickle

Navigation Menu

Repository navigation

Difference in behavior between pure Python pickle and C extension _pickle when unpickling instance of instance of metaclass #105250

Bug report

MWE

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Difference in behavior between pure Python pickle and C extension _pickle when unpickling instance of instance of metaclass #105250

Description

Bug report

MWE

Linked PRs

Activity

chaerrypick01 commented on Aug 16, 2026

corona10 commented on Aug 17, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions