Repository navigation
3.12.0b1 includes backwards incompatible change to operation of super() #105035
Description
Bug report
The optimisation to LOAD_SUPER_ATTR introduced by #104270 appears to have introduced a backwards incompatibility.
The following code will reproduce the problem:
class MyInstance:
def __new__(cls, ptr, name, bases, attrs):
self = super().__new__(cls, name, bases, attrs)
print(f"{MyInstance=} {self=} {type(self)=}, {super(MyInstance, type(self))=}")
super(MyInstance, type(self)).__setattr__(self, "ptr", ptr)
return self
def __setattr__(self, name, value):
raise Exception()
class MyClass(MyInstance, type):
def __new__(cls, name):
self = super().__new__(cls, id(name), name, (MyInstance,), {})
return self
class1 = MyClass("Class1")
print(f"{class1.ptr=}")
class2 = MyClass("Class2")
print(f"{class2.ptr=}")
Under 3.12.0a7 (and previous stable Python versions going back to at least 3.7), this will succeed, outputting:
MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class1'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
class1.ptr=4364457392
MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class2'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
class2.ptr=4365761904
Under 3.12.0b1, it raises an error:
MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class1'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
class1.ptr=4370144336
MyInstance=<class '__main__.MyInstance'> self=<class '__main__.Class2'> type(self)=<class '__main__.MyClass'>, super()=<super: <class 'MyInstance'>, <MyClass object>>
Traceback (most recent call last):
File "/Users/rkm/beeware/rubicon/objc/repro.py", line 24, in <module>
class2 = MyClass("Class2")
^^^^^^^^^^^^^^^^^
File "/Users/rkm/beeware/rubicon/objc/repro.py", line 16, in __new__
self = super().__new__(cls, id(name), name, (MyInstance,), {})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rkm/beeware/rubicon/objc/repro.py", line 7, in __new__
super().__setattr__(self, "ptr", ptr)
TypeError: expected 2 arguments, got 3
That is - the first MyClass instance can be instantiated; however, the second instance fails in mid-construction when invoking __setattr__. The state of the objects prior to the __setattr__ invocation appear to be identical, but the __setattr__ method behaves differently on the second invocation.
Git bisect has narrowed the cause down to #104270 (CC @carljm, @markshannon).
The reproduction case also fails if the call to super:
super(MyInstance, type(self)).__setattr__(self, "ptr", ptr)
is replaced with the simpler:
super().__setattr__(self, "ptr", ptr)
Background
This test case has been extracted from rubicon-objc, causing beeware/rubicon-objc#313. Rubicon is a wrapper around the Objective C runtime used by macOS and iOS; MyInstance is an analog of ObjCInstance, and MyClass is a an analog of ObjCClass. ObjCInstance has an implementation of __setattr__ to redirect attribute access to the underlying ObjC calls. However, during construction, ObjCInstance needs to store a ptr of the underlying ObjC instance. This isn't a valid ObjC attribute, so super() is used to access the underlying __setattr__ implementation to set the attribute.
Your environment
- CPython versions tested on: 3.7.9, 3.10.11, 3.12.0a7, 3.12.0b1
- Operating system and architecture: macOS ARM64
Activity
sunmy2019 commentedon May 28, 2023
Reduced test case
sunmy2019 commentedon May 28, 2023
Also, note the following passes. So it's definitely related to the interpreter core.
sunmy2019 commentedon May 28, 2023
The C call frame when the
TypeErroris raisedDetails
From the C side, it seems that
the first is called with
("call1", id(...))the second is called with
(self, "call2", id(...))sunmy2019 commentedon May 28, 2023
The comment of
_PyEval_EvalFrameDefaultsaysAnd confirmed that in the latter case, the stack is
But it should be NULL?
We can dis this function, and we get
must have something to do with this.
carljm commentedon May 30, 2023
Thanks @freakboy3742, and @sunmy2019 for the simplified repro! I've identified the issue and working on a PR to fix.
Just for the record (and only because it confused me a bit until I double-checked), my testing shows that this wasn't broken by #104270, it was broken by the original introduction of
LOAD_SUPER_ATTRin #103497.pythongh-105035: fix super() calls on unusual types (e.g. meta-types)
gh-105035: fix super() calls on unusual types (e.g. meta-types) (#105094
pythongh-105035: fix super() calls on unusual types (e.g. meta-types) (…
[3.12] gh-105035: fix super() calls on unusual types (e.g. meta-types) (
carljm commentedon May 30, 2023
This should be fixed now, in both 3.12 and main. Thanks for the report!
freakboy3742 commentedon May 30, 2023
Confirming this is now working for me. Thanks for the fast turnaround, @carljm!