Skip to content

3.12.0b1 includes backwards incompatible change to operation of super() #105035

Closed
Closed
3.12.0b1 includes backwards incompatible change to operation of super()#105035
Assignees
carljm
Labels
3.12only security fixes3.13only security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

Description

@freakboy3742
freakboy3742
opened on May 28, 2023

Bug report

The optimisation to LOAD_SUPER_ATTR introduced by #104270 appears to have introduced a backwards incompatibility.

The following code will reproduce the problem:

class MyInstance:
    def __new__(cls, ptr, name, bases, attrs):
        self = super().__new__(cls, name, bases, attrs)
        print(f"{MyInstance=} {self=} {type(self)=}, {super(MyInstance, type(self))=}")
        super(MyInstance, type(self)).__setattr__(self, "ptr", ptr)
        return self

    def __setattr__(self, name, value):
        raise Exception()


class MyClass(MyInstance, type):
    def __new__(cls, name):
        self = super().__new__(cls, id(name), name, (MyInstance,), {})

        return self


class1 = MyClass("Class1")
print(f"{class1.ptr=}")

class2 = MyClass("Class2")
print(f"{class2.ptr=}")

Under 3.12.0a7 (and previous stable Python versions going back to at least 3.7), this will succeed, outputting:

MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class1'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
class1.ptr=4364457392
MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class2'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
class2.ptr=4365761904

Under 3.12.0b1, it raises an error:

MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class1'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
class1.ptr=4370144336
MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class2'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
Traceback (most recent call last):
  File "/Users/rkm/beeware/rubicon/objc/repro.py", line 24, in <module>
    class2 = MyClass("Class2")
             ^^^^^^^^^^^^^^^^^
  File "/Users/rkm/beeware/rubicon/objc/repro.py", line 16, in __new__
    self = super().__new__(cls, id(name), name, (MyInstance,), {})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rkm/beeware/rubicon/objc/repro.py", line 7, in __new__
    super().__setattr__(self, "ptr", ptr)
TypeError:  expected 2 arguments, got 3

That is - the first MyClass instance can be instantiated; however, the second instance fails in mid-construction when invoking __setattr__. The state of the objects prior to the __setattr__ invocation appear to be identical, but the __setattr__ method behaves differently on the second invocation.

Git bisect has narrowed the cause down to #104270 (CC @carljm, @markshannon).

The reproduction case also fails if the call to super:

        super(MyInstance, type(self)).__setattr__(self, "ptr", ptr)

is replaced with the simpler:

        super().__setattr__(self, "ptr", ptr)

Background

This test case has been extracted from rubicon-objc, causing beeware/rubicon-objc#313. Rubicon is a wrapper around the Objective C runtime used by macOS and iOS; MyInstance is an analog of ObjCInstance, and MyClass is a an analog of ObjCClass. ObjCInstance has an implementation of __setattr__ to redirect attribute access to the underlying ObjC calls. However, during construction, ObjCInstance needs to store a ptr of the underlying ObjC instance. This isn't a valid ObjC attribute, so super() is used to access the underlying __setattr__ implementation to set the attribute.

Your environment

  • CPython versions tested on: 3.7.9, 3.10.11, 3.12.0a7, 3.12.0b1
  • Operating system and architecture: macOS ARM64

Linked PRs

Activity

  1. freakboy3742
    added
    type-bugAn unexpected behavior, bug, or error
    on May 28, 2023

  2. sunmy2019 commented on May 28, 2023

    @sunmy2019
    sunmy2019
    on May 28, 2023
    Member

    Reduced test case

    class MyInstance(type):
    pass


def test(name):
    self = MyInstance(name, (MyInstance,), {})
    super(MyInstance, type(self)).__setattr__(self, name, 1)


test("call1")
test("call2")  # <-- only at the second call

  3. sunmy2019 commented on May 28, 2023

    @sunmy2019
    sunmy2019
    on May 28, 2023
    Member

    Also, note the following passes. So it's definitely related to the interpreter core.

    class MyInstance(type):
    pass


def test(name):
    self = MyInstance(name, (MyInstance,), {})

    m = super(MyInstance, type(self)).__setattr__
    m(self, name, id(name))


test("call1")
test("call2")

  4. sunmy2019 commented on May 28, 2023

    @sunmy2019
    sunmy2019
    on May 28, 2023
    Member

    The C call frame when the TypeError is raised

    Details 

    #0  PyErr_Format (exception=0x555555bc11c0 <_PyExc_TypeError>, format=format@entry=0x555555a2b730 "%.200s expected %s%zd argument%s, got %zd") at Python/errors.c:1193
#1  0x000055555589f2ae in _PyArg_CheckPositional (name=name@entry=0x5555559cb7a6 "", nargs=nargs@entry=3, min=min@entry=2, max=max@entry=2) at Python/getargs.c:2813
#2  0x000055555589f344 in unpack_stack (args=0x7ffff7a23848, nargs=3, name=name@entry=0x5555559cb7a6 "", min=min@entry=2, max=max@entry=2, vargs=vargs@entry=0x7fffffffd8b0) at Python/getargs.c:2836
#3  0x000055555589f478 in PyArg_UnpackTuple (args=<optimized out>, name=name@entry=0x5555559cb7a6 "", min=min@entry=2, max=max@entry=2) at Python/getargs.c:2864
#4  0x0000555555796796 in wrap_setattr (self=0x555555db65d0, args=<optimized out>, wrapped=0x55555579cbd6 <type_setattro>) at Objects/typeobject.c:7972
#5  0x0000555555713412 in wrapperdescr_raw_call (kwds=0x0, args=0x7ffff7a23830, self=0x555555db65d0, descr=0x7ffff7b4cb90) at Objects/descrobject.c:537
#6  wrapperdescr_call (descr=descr@entry=0x7ffff7b4cb90, args=0x7ffff7a23830, args@entry=0x7ffff7a78470, kwds=kwds@entry=0x0) at Objects/descrobject.c:574
#7  0x0000555555702422 in _PyObject_MakeTpCall (tstate=tstate@entry=0x555555d21cd0 <_PyRuntime+478416>, callable=callable@entry=0x7ffff7b4cb90, args=args@entry=0x7ffff7ae50f0, nargs=<optimized out>, keywords=keywords@entry=0x0)
    at Objects/call.c:240
#8  0x0000555555702856 in _PyObject_VectorcallTstate (tstate=0x555555d21cd0 <_PyRuntime+478416>, callable=0x7ffff7b4cb90, args=0x7ffff7ae50f0, nargsf=<optimized out>, kwnames=0x0) at ./Include/internal/pycore_call.h:90
#9  0x0000555555702882 in PyObject_Vectorcall (callable=callable@entry=0x7ffff7b4cb90, args=args@entry=0x7ffff7ae50f0, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:325
#10 0x0000555555865909 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555d21cd0 <_PyRuntime+478416>, frame=0x7ffff7ae5090, throwflag=throwflag@entry=0) at Python/bytecodes.c:2643
#11 0x000055555586d3aa in _PyEval_EvalFrame (throwflag=0, frame=<optimized out>, tstate=0x555555d21cd0 <_PyRuntime+478416>) at ./Include/internal/pycore_ceval.h:87
#12 _PyEval_Vector (tstate=tstate@entry=0x555555d21cd0 <_PyRuntime+478416>, func=func@entry=0x7ffff7a184d0, locals=locals@entry=0x7ffff7a21250, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0) at Python/ceval.c:1610
#13 0x000055555586d468 in PyEval_EvalCode (co=co@entry=0x7ffff7ba9470, globals=globals@entry=0x7ffff7a21250, locals=locals@entry=0x7ffff7a21250) at Python/ceval.c:567
#14 0x00005555558decdf in run_eval_code_obj (tstate=tstate@entry=0x555555d21cd0 <_PyRuntime+478416>, co=co@entry=0x7ffff7ba9470, globals=globals@entry=0x7ffff7a21250, locals=locals@entry=0x7ffff7a21250) at Python/pythonrun.c:1695
#15 0x00005555558df43f in run_mod (mod=mod@entry=0x555555dbff20, filename=filename@entry=0x7ffff7a22560, globals=globals@entry=0x7ffff7a21250, locals=locals@entry=0x7ffff7a21250, flags=flags@entry=0x7fffffffdea8, arena=arena@entry=0x7ffff7a63ee0)
    at Python/pythonrun.c:1716
#16 0x00005555558df54f in pyrun_file (fp=fp@entry=0x555555d3cd20, filename=filename@entry=0x7ffff7a22560, start=start@entry=257, globals=globals@entry=0x7ffff7a21250, locals=locals@entry=0x7ffff7a21250, closeit=closeit@entry=1, 
    flags=0x7fffffffdea8) at Python/pythonrun.c:1616
#17 0x00005555558e2c7d in _PyRun_SimpleFileObject (fp=fp@entry=0x555555d3cd20, filename=filename@entry=0x7ffff7a22560, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdea8) at Python/pythonrun.c:433
#18 0x00005555558e2ed8 in _PyRun_AnyFileObject (fp=fp@entry=0x555555d3cd20, filename=filename@entry=0x7ffff7a22560, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdea8) at Python/pythonrun.c:78
#19 0x000055555590c13a in pymain_run_file_obj (program_name=program_name@entry=0x7ffff7a63d60, filename=filename@entry=0x7ffff7a22560, skip_source_first_line=0) at Modules/main.c:360
#20 0x000055555590c4d9 in pymain_run_file (config=config@entry=0x555555d04008 <_PyRuntime+356360>) at Modules/main.c:379
#21 0x000055555590d656 in pymain_run_python (exitcode=exitcode@entry=0x7fffffffe024) at Modules/main.c:610
#22 0x000055555590d6c6 in Py_RunMain () at Modules/main.c:689
#23 0x000055555590d73d in pymain_main (args=args@entry=0x7fffffffe080) at Modules/main.c:719
#24 0x000055555590d800 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:743
#25 0x00005555556537e6 in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15

    From the C side, it seems that
    the first is called with ("call1", id(...))
    the second is called with (self, "call2", id(...))

  5. sunmy2019 commented on May 28, 2023

    @sunmy2019
    sunmy2019
    on May 28, 2023
    Member

    The comment of _PyEval_EvalFrameDefault says

            // On entry, the stack is either
        //   [NULL, callable, arg1, arg2, ...]
        // or
        //   [method, self, arg1, arg2, ...]

    And confirmed that in the latter case, the stack is

            //   [method, self, arg1, arg2, ...]

    But it should be NULL?

    We can dis this function, and we get

      5           0 RESUME                   0

  6           2 LOAD_GLOBAL              1 (NULL + MyInstance)
             12 LOAD_FAST                0 (name)
             14 LOAD_GLOBAL              0 (MyInstance)
             24 BUILD_TUPLE              1
             26 BUILD_MAP                0
             28 CALL                     3
             36 STORE_FAST               1 (self)

 12          38 LOAD_GLOBAL              2 (super)
             48 LOAD_GLOBAL              0 (MyInstance)
             58 LOAD_GLOBAL              5 (NULL + type)
             68 LOAD_FAST                1 (self)
             70 CALL                     1
             78 LOAD_SUPER_ATTR         15 (NULL|self + __setattr__)
             82 LOAD_FAST                1 (self)
             84 LOAD_FAST                0 (name)
             86 LOAD_CONST               1 (1)
             88 CALL                     3
             96 POP_TOP
             98 RETURN_CONST             0 (None)

                 78 LOAD_SUPER_ATTR         15 (NULL|self + __setattr__)

    must have something to do with this.

  6. sunmy2019
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    3.12only security fixes
    3.13only security fixes
    on May 28, 2023
  7. Fidget-Spinner
    assigned
    carljm
    on May 28, 2023

  8. carljm commented on May 30, 2023

    @carljm
    carljm
    on May 30, 2023
    Member

    Thanks @freakboy3742, and @sunmy2019 for the simplified repro! I've identified the issue and working on a PR to fix.

    Just for the record (and only because it confused me a bit until I double-checked), my testing shows that this wasn't broken by #104270, it was broken by the original introduction of LOAD_SUPER_ATTR in #103497.

  9. carljm
    added 2 commits that reference this issue on May 30, 2023

    pythongh-105035: fix super() calls on unusual types (e.g. meta-types)

    4ef2316

    gh-105035: fix super() calls on unusual types (e.g. meta-types) (#105094

    68c75c3
  10. miss-islington
    added a commit that references this issue on May 30, 2023

    pythongh-105035: fix super() calls on unusual types (e.g. meta-types) (…

    8eef8d4
  11. carljm
    added a commit that references this issue on May 30, 2023

    [3.12] gh-105035: fix super() calls on unusual types (e.g. meta-types) (

    5fff491

  12. carljm commented on May 30, 2023

    @carljm
    carljm
    on May 30, 2023
    Member

    This should be fixed now, in both 3.12 and main. Thanks for the report!

  13. carljm
    closed this as completedon May 30, 2023

  14. freakboy3742 commented on May 30, 2023

    @freakboy3742
    freakboy3742
    on May 30, 2023
    ContributorAuthor

    Confirming this is now working for me. Thanks for the fast turnaround, @carljm!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.12only security fixes3.13only security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions