Allow __slots__ on classes with Py_TPFLAGS_ITEMS_AT_END

@encukou

Bug report

If a base class is a PyVarObject, classes inheriting from that base cannot add __dict__ or __weakref__ through __slots__ declaration on the class. If __slots__ is not empty, the following exception is raised:
TypeError: nonempty __slots__ not supported for subtype of '...'

Assume that we have a PyVarObject foo.FooBase class (implementation below). Trying to inherit from the class and requesting weakref support doesn't work.

import foo
class WithWeakrefAndDict(foo.FooBase):
    __slots__ = ('__weakref__', '__dict__')

`foo` module implementation

#define PY_SSIZE_T_CLEAN
#include <Python.h>

static const Py_ssize_t N_EXTRA = 4;

typedef struct {
    PyObject_VAR_HEAD
} FooBase;

static PyObject **
FooBase_get_storage(PyObject *self)
{
    char *addr = (char *)self;
    return (PyObject **)(addr + Py_TYPE(self)->tp_basicsize);
}

static int
FooBase_traverse(PyObject *self, visitproc visit, void *arg)
{
    PyObject **storage = FooBase_get_storage(self);
    for (int i = 0; i < N_EXTRA; i++) {
        Py_VISIT(storage[i]);
    }
    Py_VISIT(Py_TYPE(self));
    return 0;
}

static int
FooBase_clear(PyObject *self)
{
    PyObject **storage = FooBase_get_storage(self);
    for (int i = 0; i < N_EXTRA; i++) {
        Py_CLEAR(storage[i]);
    }
    return 0;
}

static void
FooBase_dealloc(PyObject *self)
{
    PyTypeObject *tp = Py_TYPE(self);
    PyObject_GC_UnTrack(self);
    FooBase_clear(self);
    PyObject_GC_Del(self);
    if (tp->tp_flags & Py_TPFLAGS_HEAPTYPE)
        Py_DECREF(tp);
}

static PyObject *
FooBase_new(PyTypeObject *type, PyObject *args, PyObject *kwargs)
{
    PyVarObject *obj = PyObject_GC_NewVar(PyVarObject, type, N_EXTRA);
    if (obj == NULL)
        return NULL;
    PyObject **storage = FooBase_get_storage((PyObject *)obj);
    for (int i = 0; i < Py_SIZE(obj); i++)
        storage[i] = NULL;

    PyObject_GC_Track(obj);
    return (PyObject *)obj;
}

static PyObject *
FooBase_get_extra(PyObject *self, PyObject *args)
{
    Py_ssize_t idx;
    if (!PyArg_ParseTuple(args, "n", &idx))
        return NULL;
    if (idx < 0 || idx >= N_EXTRA) {
        PyErr_Format(PyExc_ValueError, "idx must be >= 0 and < %zd", N_EXTRA);
        return NULL;
    }

    PyObject **storage = FooBase_get_storage(self);
    PyObject *value = storage[idx];
    if (!value) {
        Py_RETURN_NONE;
    } else {
        Py_INCREF(value);
        return value;
    }
}

static PyObject *
FooBase_set_extra(PyObject *self, PyObject *args)
{
    Py_ssize_t idx;
    PyObject *value;
    if (!PyArg_ParseTuple(args, "nO", &idx, &value))
        return NULL;
    if (idx < 0 || idx >= N_EXTRA) {
        PyErr_Format(PyExc_ValueError, "idx must be >= 0 and < %zd", N_EXTRA);
        return NULL;
    }

    PyObject **storage = FooBase_get_storage(self);
    Py_CLEAR(storage[idx]);
    Py_INCREF(value);
    storage[idx] = value;

    Py_RETURN_NONE;
}

static PyMethodDef FooBase_methods[] = {
    {"get_extra", FooBase_get_extra, METH_VARARGS, NULL},
    {"set_extra", FooBase_set_extra, METH_VARARGS, NULL},
    {NULL}
};

static PyTypeObject FooBase_Type = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "foo.FooBase",
    sizeof(FooBase),
    .tp_dealloc = (destructor)FooBase_dealloc,
    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC,
    .tp_traverse = FooBase_traverse,
    .tp_clear = FooBase_clear,
    .tp_methods = FooBase_methods,
    .tp_new = FooBase_new,
    .tp_itemsize = sizeof(PyObject *)
};

static struct PyModuleDef foo_module = {
    PyModuleDef_HEAD_INIT,
    "foo",
    NULL,
    -1,
};

PyMODINIT_FUNC
PyInit_foo(void)
{
    PyObject *m = PyModule_Create(&foo_module);
    if (!m)
        return NULL;
    if (PyType_Ready(&FooBase_Type) < 0)
        return NULL;
    Py_INCREF(&FooBase_Type);
    PyModule_AddObject(m, "FooBase", (PyObject *)&FooBase_Type);
    return m;
}

Your environment

CPython versions tested on: 3.11.3
Operating system and architecture: Linux, x86_64

CC: @encukou

Linked PRs

This is a duplicate of #61497. The isn't isn't so much that these particular slots aren't supported, but that no slots are supported at all var these objects.

Let's use this issue for enabling this with Py_TPFLAGS_ITEMS_AT_END specifically. I plan to look into it for 3.13.

This particular issue (support __dict__ and __weakref__) can be solved relatively easy. Currently variable length types can have either both __dict__ and __weakref__, or none of them, so it is possible.

But I think that we should support also general slots, there are use cases for this. So this issue can be a part of #41779.

pythongh-103740: Improve support of __slots__ for variable-length cla…

#141636 implements two features:

Support __dict__ and __weakref__ slots for any class.
Support any __slots__ for subclasses of type and classes with Py_TPFLAGS_ITEMS_AT_END.

If you with, I can split it on two PRs, if this will not slow down review.

My next step is to support any __slots__ for subclasses of tuple. And then we can plan support any variable-length types (it is not easy).

pythongh-103740: Improve support of __slots__ for variable-length cla…

pythongh-103740: Allow defining the __dict__ and __weakref__ __slots_…

pythongh-103740: Support any slots in the type subclasses

Copying from the PR:

The only Py_TPFLAGS_ITEMS_AT_END type in stdlib is type itself.
So, this PR allows slotted types. Adding data descriptors to types leads to some rather hard-to-explain behaviour, for example:

>>> class Meta(type):
...     __slots__ = ['foo']  # now possible
...     
>>> class X(metaclass=Meta):
...     def foo(self): return 'f'
...     
>>> X().foo
<bound method X.foo of <__main__.X object at 0x7f7c579c2900>>
>>> X.foo
Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    X.foo
AttributeError: 'Meta' object has no attribute 'foo'
>>> X.foo = slice
>>> X.foo(X())
slice(None, <__main__.X object at 0x7f7c5788c050>, None)
>>> X().foo()
'f'

Do we want to make this easier?

Py_TPFLAGS_ITEMS_AT_END, this feature looks to be specifically about allowing descriptors for type, so I'm not sure we want it.

What about explicitly disabling arbitrary __slots__ on type subclasses, given how many type optimizations assume that the __dict__ is authoritative?

I do not understand what problems can be caused by adding support of __slots__ in the type subtypes.

There are already data descriptors in type. And they can be different from data descriptors in __dict__. This is a feature. For example, functions have __doc__, and it is different from the __doc__ of the function type.
PEP 697 intentionally added support of data descriptors referring to the extended data in the type subtypes. If this was mistake, should not we forbid Py_TPFLAGS_ITEMS_AT_END for the type subtypes?

There are already data descriptors in type. And they can be different from data descriptors in __dict__. This is a feature. For example, functions have __doc__, and it is different from the __doc__ of the function type.

Yes, that is a feature, but I doubt that users expect __slots__ to behave this way.
IMO, the usual visible effect of __slots__ is to:

disable some attributes (not applicable here since type already has __dict__)
improve memory use (not too important here: we don't avoid a __dict__, and type itself is huge: you'd need many slots for them to be a significant part of a type's memory footprint.)

Descriptors are an implementation detail; and here they'd be responsible for confusing interactions with optimizations that assume attributes are in __dict__.

I can't see a use case for __slots__ on type, only downsides.

It's very different with third-party types that define Py_TPFLAGS_ITEMS_AT_END though. On something that doesn't have __dict__, or doesn't assume that data is in __dict__, __slots__ can be very useful. It's only type where they seem like a footgun.

PEP 697 intentionally added support of data descriptors referring to the extended data in the type subtypes.

PEP 697 was primarily about adding a C-level chunk of memory, not about Python attributes. It's true that PEP 697 also added a way to expose the contents of that memory to Python, but that wasn't the main point -- and it definitely was not specifically for type.
Defining attributes is very different in C than in Python.

Allowing __slots__ in classes with the Py_TPFLAGS_ITEMS_AT_END flag will not add anything principally new. It will just enable in Python what is already allowed in C (but in more limited form). It will just remove unnecessary restriction. If there is any issue with this for type subclasses, we should disallow creating such descriptors in C. But I do not think there will be any new issue, because standard data descriptors existed in type for years.

doesn't assume that data is in __dict__

This assumption never was correct.

>>> def f(): 'docstring'
... 
>>> f.__doc__
'docstring'
>>> '__doc__' in f.__dict__
False
>>> type(f).__doc__
'Create a function object.\n\n  code\n    a code object\n  globals\n    the globals dictionary\n  name\n    a string that overrides the name from the code object\n  argdefs\n    a tuple that specifies the default argument values\n  closure\n    a tuple that supplies the bindings for free variables\n  kwdefaults\n    a dictionary that specifies the default keyword argument values'

I only want consistency. Either allow __slots__ in types with Py_TPFLAGS_ITEMS_AT_END, or disallow data descriptors referring to the data added in subclasses with Py_TPFLAGS_ITEMS_AT_END. Either allow __slots__ in types with Py_TPFLAGS_ITEMS_AT_END except the type subclasses, or disallow descriptors for the new data in the type subclasses.

This assumption never was correct.

Right, in general it wasn't.
I don't think __doc__ is a good example -- it has its own specialized descriptor to ensure it's not inherited. It doesn't work like normal attributes.
Allowing descriptors on type make it possible to add functionality like that. That's good; I wouldn't disallow them entirely.
But with __slots__, the way they're defined automatically from Python code makes them confusing. Combined with the fact that I can see no good use case for __slots__ on type, I'm reluctant to add this new feature. (You can already add Python attributes to classes -- it's different from exposing C data by PEP 697.)

I think the discussion is going in circles. Do you want to open a Discourse topic to get additional opinions?

Navigation Menu

Repository navigation

Allow slots on classes with Py_TPFLAGS_ITEMS_AT_END #103740

Bug report

Your environment

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Allow __slots__ on classes with Py_TPFLAGS_ITEMS_AT_END #103740

Description

Bug report

Your environment

Linked PRs

Activity

ronaldoussoren commented on Apr 24, 2023

encukou commented on Apr 24, 2023

serhiy-storchaka commented on Dec 30, 2024

serhiy-storchaka commented on Nov 16, 2025

encukou commented on Nov 20, 2025

encukou commented on Nov 20, 2025

serhiy-storchaka commented on Nov 20, 2025

encukou commented on Nov 21, 2025

serhiy-storchaka commented on Nov 21, 2025

encukou commented on Nov 24, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Allow slots on classes with Py_TPFLAGS_ITEMS_AT_END #103740