Repository navigation
Breaking backward compatibility between ctypes and metaclasses in Python 3.13. #124520
Description
Activity
Thanks for reporting this! If I knew about this earlier I'd try to make your life easier, but at this point it looks like a docs change is the best way to go.
I'm afraid that by using
type(c_void_p)to get toctypes.PyCSimpleTypeinternals, and by importing_pointer_type_cache, you're getting in the realm of things that can change.
Unfortunately,ctypeshas been... stable? neglected? ... for a long time, and these workarounds are common in the wild.
If you can, please do plan to test with 3.14 as the alphas/betas come out, and ping me on any other incompatibilities you might find.For the diff you linked:
Unfortunately I don't have a Windows box to test on. What error are you getting if you put all of the logic in
__init__? (Is it__init__() should return None? Don't returnselffrom__init__.)If the if/else is needed, I recommend putting the logic in common methods like
_newand_initto remove the duplication, so the branch becomes something like:if sys.version_info >= (3, 13): def __new__(cls, ...): return cls._new(...) def __init__(self, ...): self._init(...) else: def __new__(cls, ...): self = cls._new(cls, ...) self.__init__(...) return self
or even:
if sys.version_info >= (3, 13): __new__ = _new __init__ = _init else: def __new__(cls, ...): self = cls._new(cls, ...) self._init(...) return self
Reacted by Jun KomodaIt is necessary to pass arguments directly to the metaclass instead of using new in places where the metaclass is instantiated (i.e., where the class is dynamically defined).
Hm, I don't understand this point. Do you have an example?
It is necessary to pass arguments directly to the metaclass instead of using new in places where the metaclass is instantiated (i.e., where the class is dynamically defined).
Hm, I don't understand this point. Do you have an example?
In
comtypes, the following part corresponds to this.https://github.com/enthought/comtypes/compare/c631f97..6f036d4
meta = type(_safearray.tagSAFEARRAY) - sa_type = meta.__new__( - meta, "SAFEARRAY_%s" % itemtype.__name__, (_safearray.tagSAFEARRAY,), {} - ) + sa_type = meta(f"SAFEARRAY_{itemtype.__name__}", (_safearray.tagSAFEARRAY,), {})
Reacted by Petr ViktorinUnfortunately I don't have a Windows box to test on. What error are you getting if you put all of the logic in
__init__?--- a/comtypes/_post_coinit/unknwn.py +++ b/comtypes/_post_coinit/unknwn.py @@ -75,7 +75,9 @@ class _cominterface_meta(type): new_cls._methods_ = methods if dispmethods is not None: new_cls._disp_methods_ = dispmethods + return new_cls + def __init__(self, name, bases, namespace): # If we sublass a COM interface, for example: # # class IDispatch(IUnknown): @@ -85,23 +87,22 @@ class _cominterface_meta(type): # subclass of POINTER(IUnknown) because of the way ctypes # typechecks work. if bases == (object,): - _ptr_bases = (new_cls, _compointer_base) + _ptr_bases = (self, _compointer_base) else: - _ptr_bases = (new_cls, POINTER(bases[0])) + _ptr_bases = (self, POINTER(bases[0])) # The interface 'new_cls' is used as a mixin. p = type(_compointer_base)( - "POINTER(%s)" % new_cls.__name__, + "POINTER(%s)" % self.__name__, _ptr_bases, - {"__com_interface__": new_cls, "_needs_com_addref_": None}, + {"__com_interface__": self, "_needs_com_addref_": None}, ) from ctypes import _pointer_type_cache - _pointer_type_cache[new_cls] = p - - if new_cls._case_insensitive_: + _pointer_type_cache[self] = p + if self._case_insensitive_: @patcher.Patch(p) class CaseInsensitive(object): # case insensitive attributes for COM methods and properties @@ -155,8 +156,6 @@ class _cominterface_meta(type): CopyComPointer(value, self) - return new_cls -
The following error occurs in Python 3.11 when making the changes mentioned above.
Traceback (most recent call last): File "...\Python311\Lib\unittest\__main__.py", line 18, in <module> main(module=None) File "...\Python311\Lib\unittest\main.py", line 101, in __init__ self.parseArgs(argv) File "...\Python311\Lib\unittest\main.py", line 150, in parseArgs self.createTests() File "...\Python311\Lib\unittest\main.py", line 161, in createTests self.test = self.testLoader.loadTestsFromNames(self.testNames, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "...\Python311\Lib\unittest\loader.py", line 220, in loadTestsFromNames suites = [self.loadTestsFromName(name, module) for name in names] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "...\Python311\Lib\unittest\loader.py", line 220, in <listcomp> suites = [self.loadTestsFromName(name, module) for name in names] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "...\Python311\Lib\unittest\loader.py", line 154, in loadTestsFromName module = __import__(module_name) ^^^^^^^^^^^^^^^^^^^^^^^ from comtypes._post_coinit import _shutdown File "...\comtypes\_post_coinit\__init__.py", line 15, in <module> from comtypes._post_coinit.unknwn import _shutdown # noqa ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "...\comtypes\_post_coinit\unknwn.py", line 370, in <module> class _compointer_base(c_void_p, metaclass=_compointer_meta): File "...\comtypes\_post_coinit\unknwn.py", line 95, in __init__ p = type(_compointer_base)( ^^^^^^^^^^^^^^^^ NameError: name '_compointer_base' is not defined. Did you mean: '_compointer_meta'?Thanks!
I won't be able to debug that by reading the code. I'll follow up next week when I get back to a Windows PC.Reacted by Jun Komoda and Gregory P. SmithThank you.
If I knew about this earlier I'd try to make your life easier, but at this point it looks like a docs change is the best way to go.
Fortunately, libraries like
comtypesandpygletwere able to adapt with changes to the codebase.
However, if there is a codebase that performs more complex pre-processing before callingsuper().__new__within__new__, it might need more complex workarounds.If it is still possible to modify things so that performing all initialization in
__new__does not result in an error, that would be great.
Alternatively, documenting a workaround for such cases would be helpful for maintainers of those kinds of packages.- added a commit that references this issue
on Sep 27, 2024 10 remaining items
Hi,
Thank you for your investigation.
_initializing_coclass_meta = False class _coclass_meta(type): def __init__(self, name, bases, namespace): ... global _initializing_coclass_meta if _initializing_coclass_meta: return try: _initializing_coclass_meta = True PTR = _coclass_pointer_meta( "POINTER(%s)" % self.__name__, (self, c_void_p), { "__ctypes_from_outparam__": _wrap_coclass, "from_param": classmethod(_coclass_from_param), }, ) finally: _initializing_coclass_meta = False ...
…and similarly in
_cominterface_meta.Instead of duplicating the code, you can define
__new__for 3.11 but__init__for 3.13 like this:class _cominterface_meta(type): def __init(self, name, bases, namespace): # (put all the init code here) if sys.version_info < (3, 13): def __new__(cls, name, bases, namespace): self = type.__new__(cls, name, bases, namespace) self.__init(name, bases, namespace) return self else: __init__ = __init
it definitely worked in Python 3.13 and earlier versions.
(I modified the version bridge to useif sys.version_info >= (3, 13):.)
(You can see also enthought/comtypes@00cf639^...3ef727c, https://github.com/enthought/comtypes/actions/runs/11183030620)But,
This looks ugly, but IMO it better expresses that a part of the usual initialazation is skipped (and why).
(There might be a better way to detect the recursion?)As you mentioned, using
globalandtry...finally...to modify a module-level signal variable from within a method seems like stranger code compared to defining_newand_initand switching how they are called using a version bridge (like in enthought/comtypes@6c24dcf^...ea9d9fc).Until we come up with a more elegant way to signal the stopping of recursion, I believe it might be better to add a comment like
# This stops excessive recursionin addition to the implementation like enthought/comtypes@6c24dcf^...ea9d9fc.I would be happy to hear any further thoughts you may have.
Reacted by Petr ViktorinI investigated what values are assigned to
self,name,bases, andnamespacewhen using the global variable_initializing_..._metaas a signal for early returns.
Based on that, I modified the codebases to perform early returns depending on which subclass/instanceselfis and the components ofbases, which worked well.
This approach eliminates the need to define__init__and version bridges.These changes have been committed and pushed to enthought/comtypes@4c779617^...312e268 (https://github.com/junkmd/comtypes/tree/py313_early_returns).
This change makes explicit what the metaclass would have done implicitly until now.
Since it is not intuitive and requires multiple code jumps to follow the flow of processing, I added some comments to aid understanding.The changes using version bridges also has been committed to enthought/comtypes@4c779617^...6501fd1 (https://github.com/junkmd/comtypes/tree/py313_version_bridges).
Any opinions would be appreciated.
That sounds reasonable.
My main point was that you only want to skip only the recursion (i.e. registering the pointer class), but not any other step of the initialization. Doing that by skipping an entire__new__or__init__of a particular base is fragile, and confusing to readers.
An early return before your customizations is the way to go; as for the conditions for it, you're better suited to define them :)(That said, I hope the details of the initialization won't need to change for another decade and so the fragility will be just theoretical from now on. Again, I'd appreciate if you test with 3.14 alphas/betas as they come out.)
One more possibility to consider would be catching the
NameErrorraised when accessing_compointer_baseearly, e.g.:try: is_compointer = issubclass(self, _compointer_base) except NameError: # On some versions of Python, `_compointer_base` is not # yet available in the accessible namespace at this # point in its initialization. # In this case, `self` will become `_compointer_base` # later, so `issubclass(self, _compointer_base)` # will be True. is_compointer = True # Double-check that self is actually _compointer_base. assert bases == (c_void_p,) if is_compointer: # `self` is `POINTER(interface)` type. # Prevent registering a pointer to a pointer (to a pointer...), # which would lead to infinite recursion. # Depending on a version or revision of Python, this may be essential. return self
(Not tested; I didn't boot Windows today.)
Reacted by Jun KomodaI'll close the issue as there's nothing more to do in CPython code/docs.
- moved this from Todo to Done in Release and Deferred blockers 🚫
on Oct 7, 2024 Thank you for your feedback.
That sounds reasonable.
My main point was that you only want to skip only the recursion (i.e. registering the pointer class), but not any other step of the initialization. Doing that by skipping an entire__new__or__init__of a particular base is fragile, and confusing to readers.
An early return before your customizations is the way to go; as for the conditions for it, you're better suited to define them :)It's very reassuring to hear something like this from the maintainer of the library.
(That said, I hope the details of the initialization won't need to change for another decade and so the fragility will be just theoretical from now on. Again, I'd appreciate if you test with 3.14 alphas/betas as they come out.)
When Python 3.14 alphas/betas is released, I plan to create a repository has a GitHub Actions workflow that will run tests of
comtypeson those versions.
I’m considering sharing those URLs to make information dissemination easier.One more possibility to consider would be catching the
NameErrorraised when accessing_compointer_baseearly, e.g.:I have confirmed that the code you proposed above also works correctly.
(enthought/comtypes@464f566, https://github.com/enthought/comtypes/actions/runs/11215471734)
However, since_com..._metaand_com..._basedefined incomtypes._post_coinit.unknwnare intended to be used only for defining classes forIUnknownorPOINTER(interface), I believe that as long as ifbasesis(c_void_p,), it should be safe to assume thatselfis_compointer_base. Thetry...except...for theNameErrorseemed a bit excessive to me.
That said, your comments on the code base would be very helpful for readers in understanding the process flow whether with or without catching theNameError. I will definitely take them into account. Thank you.Reacted by Petr ViktorinI'll close the issue as there's nothing more to do in CPython code/docs.
Thank you very much again.
I am also considering submitting another issue/PR to
cpythonto add tests to ensure that no regressions occur in metaclass implementations similar to those used incomtypes.
If you have any thoughts, I'd love to hear them.I am also considering submitting another issue/PR to cpython to add tests to ensure that no regressions occur in metaclass implementations similar to those used in comtypes.
Thank you!
You can have them refer to this closed issue (gh-124520:). If you send a PR, please comment here so I'll notice it.Reacted by Jun KomodaI have created a PoC repository to verify whether
comtypesworks with development or pre-release versions of Python.
https://github.com/junkmd/comtypes-python-dev-compatibility/In the above repository, I have confirmed that the combination of
comtypesand3.14.0-alpha.1works.
https://github.com/junkmd/comtypes-python-dev-compatibility/actions/runs/11438904158/job/31821475344I am also considering submitting another issue/PR to cpython to add tests to ensure that no regressions occur in metaclass implementations similar to those used in comtypes.
Thank you! You can have them refer to this closed issue (
gh-124520:). If you send a PR, please comment here so I'll notice it.Please see this: #125783
Reacted by Petr Viktorin- added a commit that references this issue
on Aug 26, 2025 - added a commit that references this issue
on May 13, 2026
Documentation
Background
I am one of the maintainers of
comtypes.comtypesis based onctypesand uses metaclasses to implementIUnknown.It was reported to the
comtypescommunity that an error occurs when attempting to use conventional metaclasses with Python 3.13.A similar error was encountered in
pyglet, which also uses metaclasses to implement COM interfaces, when running on Python 3.13.By referring to pyglet/pyglet#1196 and pyglet/pyglet#1199, I made several modifications to the code through trial and error, and now
comtypesworks in both Python 3.13 and earlier versions without problems:__new__in places where the metaclass is instantiated (i.e., where the class is dynamically defined). This also works in versions prior to Python 3.13.type.__new__, any remaining initialization would be handled in__init__instead of in__new__. Since this results in an error in versions prior to Python 3.13, a bridge usingsys.version_infois necessary.I think these changes are likely related to #114314 and #117142 and were introduced by the PRs linked to those issues.
Since this change to
ctypesbreaks compatibility, I think it should be mentioned in the What’s New In Python 3.13 and/or in thectypesdocumentation.There are likely other projects besides
comtypesandpygletthat rely on the combination ofctypesand metaclasses, and I want to prevent confusion for those maintainers when they try to support Python 3.13.(Additionally, I would like to ask with the
ctypesmaintainers to confirm whether the changes for the metaclasses incomtypes(andpyglet) are appropriate.)Linked PRs