Vectorcall optimization can bypass a custom metaclass's __call__

[cocolato]

Bug report

The vectorcall optimization for class calls can skip a custom metaclass's __call__. This happens when a call site first specializes on a built-in class, then receives a class with a custom metaclass.

Reproduced with msgspec==0.21.1:

import msgspec


class ValidatedAmount(msgspec.StructMeta):
    def __call__(cls, value):
        cents = int(value)
        if cents <= 0:
            raise ValueError("payment amount must be positive")
        return super().__call__(cents)


class PaymentAmount(msgspec.Struct, metaclass=ValidatedAmount):
    cents: int


converters = (int, int, PaymentAmount)
row = ("42", "2", "-100")
print([convert(value) for convert, value in zip(converters, row)])

Expected:

ValueError: payment amount must be positive

Actual:

[42, 2, PaymentAmount(cents='-100')]

The affected optimization is CALL_BUILTIN_CLASS. Its runtime guard checks PyType_Check(callable) and a non-NULL tp_vectorcall. A class with a custom metaclass can pass these checks, after which the call handler invokes its tp_vectorcall directly, bypassing the metaclass's __call__.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-158788: Preserve metaclass __call__ in vectorcall optimization #158789