_py_abc Python implementation of abc is not thread-safe

Bug report

The update to the invalidation counter is not thread-safe and can lose updates in some Python implementations:

Failures seen on:

Python 3.14t
Python 3.9
pypy3.10
pypy3.11

But not on Python 3.10-3.14 with GIL due to limited GIL switch opportunities.

Lines 54 to 70 in 05e89c3

    
               def register(cls, subclass): 
        
                   """Register a virtual subclass of an ABC. 
        
                   Returns the subclass, to allow usage as a class decorator. 
        
                   """ 
        
                   if not isinstance(subclass, type): 
        
                       raise TypeError("Can only register classes") 
        
                   if issubclass(subclass, cls): 
        
                       return subclass  # Already a subclass 
        
                   # Subtle: test for cycles *after* testing for "already a subclass"; 
        
                   # this means we allow X.register(X) and interpret it as a no-op. 
        
                   if issubclass(cls, subclass): 
        
                       # This would create a cycle, which is bad for the algorithm below 
        
                       raise RuntimeError("Refusing to create an inheritance cycle") 
        
                   cls._abc_registry.add(subclass) 
        
                   ABCMeta._abc_invalidation_counter += 1  # Invalidate negative cache 
        
                   return subclass

For example, consider the following repro, adapted from test_abc.test_registration_basics:

import _py_abc as abc # Use Python implementation of ABCs!!
import threading
import os
import sys

sys.setswitchinterval(1e-6)

N = 5

def run(b):
    b.wait()

    class A(metaclass=abc.ABCMeta):
        pass
    A.register(int)
    if not isinstance(42, A):
        print("Oops!")
        os._exit(1)

def main():
    for _ in range(10000):
        threads = []
        b = threading.Barrier(N)
        for _ in range(N):
            t = threading.Thread(target=run, args=(b,))
            threads.append(t)
            t.start()
        for t in threads:
            t.join()


if __name__ == "__main__":
    main()

Linked PRs

gh-130095: Mark _py_abc tests as not thread-safe #130131

@cfbolz

I'm not really sure what to do here. We could add a threading.Lock() to protect _abc_invalidation_counter, but I'm not sure we want to import the threading module in _py_abc.py.

cc @cfbolz in case you have thoughts on this.

Not sure if this is relevant: https://github.com/python/cpython/actions/runs/13335219139/job/37248893358?pr=129175 (I don't know if the test is testing the Python or the C implementation).

EDIT: I think this is the reproducer you had. But now we caught it in the CI

pythongh-130095: Mark _py_abc tests as not thread-safe

@picnixz

Thanks @picnixz. I put up a PR to avoid running them with --parallel-threads for now until we figure out if and how to make _py_abc thread-safe.

yeah, importing threading in _py_abc leads to a circular import problem if _abc is not available. What about using itertools.count? that's supposed to be thread-safe, isn't it?

hm, doesn't quite work, because itertools.count has no way to read the current value, only a way to increase the count and get the next value.

If I'm right, the problem comes from three bytecode operations: LOAD_ATTR, BINARY_OP, and STORE_ATTR. I think there are two ways to fix this: using a specialized operation for atomic increment/update or the same built-in function. Current main branch has only three uses of _Py_atomic_add_uint64 - qsbr, _asynciomodule and _abc, so there are no existing ways to do atomic update from python. I believe the quickest way to add private built-in for this, but the right one to add atomic ops.

Sorry if I'm obvious.

The pure Python companion modules like _py_abc aren't really used by CPython in normal circumstances outside of tests. They are primarily used by other Python implementations like PyPy. I don't think it makes much sense to consider CPython specific changes to bytecode.

https://peps.python.org/pep-0399/

Thanks for the link - it is useful for my understanding.
I’m stick to atomic ops, but I’m sure you have much more and deeper understanding.

gh-130095: Mark `_py_abc` tests as not thread-safe (gh-130131)

@colesbury

@colesbury if you are specifically concerned about pypy, we don't use _py_abc.py anymore either.

Thanks for the clarification. I also noticed a failure when I substituted abc for _py_abc with pypy311, so there might be a thread-safety issue there as well.

If neither pypy nor CPython use _py_abc outside of tests, I'm not going to spend much more time on this.

Do we want to fix this or may be deprecate it (if it is not used anymore by pypy and CPython)?

Usage over 15_000 pypi packages

 python ../../misc/cpython/search_pypi_top.py -q . "_py_abc"
.\cx_freeze-8.3.0.tar.gz: cx_freeze-8.3.0/cx_Freeze/hooks/__init__.py: module.exclude_names.add("_py_abc")
.\cx_freeze-8.3.0.tar.gz: cx_freeze-8.3.0/cx_Freeze/hooks/__init__.py: finder.include_module("_py_abc")
.\cx_freeze-8.4.1.tar.gz: cx_freeze-8.4.1/cx_Freeze/hooks/__init__.py: module.exclude_names.add("_py_abc")
.\cx_freeze-8.4.1.tar.gz: cx_freeze-8.4.1/cx_Freeze/hooks/__init__.py: finder.include_module("_py_abc")
.\contrast_agent-10.11.0.tar.gz: contrast_agent-10.11.0/src/contrast_vendor/stdlib_list/lists/3.9.txt: _py_abc
.\contrast_agent-10.17.0.tar.gz: contrast_agent-10.17.0/src/contrast_vendor/stdlib_list/lists/3.9.txt: _py_abc
.\astroid-3.3.10.tar.gz: astroid-3.3.10/astroid/_backport_stdlib_names.py: "_py_abc",
.\astroid-3.3.10.tar.gz: astroid-3.3.10/tests/test_scoped_nodes.py: self.assertIn(metaclass.qname(), ("abc.ABCMeta", "_py_abc.ABCMeta"))
.\basedmypy-2.10.0.tar.gz: basedmypy-2.10.0/basedmypy.egg-info/SOURCES.txt: mypy/typeshed/stdlib/_py_abc.pyi
.\astroid-4.0.1.tar.gz: astroid-4.0.1/tests/test_scoped_nodes.py: self.assertIn(metaclass.qname(), ("abc.ABCMeta", "_py_abc.ABCMeta"))
.\basedmypy-2.10.0.tar.gz: basedmypy-2.10.0/mypy/typeshed/stdlib/VERSIONS: _py_abc: 3.7-
.\basedpyright-1.29.5.tar.gz: basedpyright-1.29.5/basedpyright/dist/typeshed-fallback/stdlib/VERSIONS: _py_abc: 3.7-
.\basedpyright-1.29.5.tar.gz: basedpyright-1.29.5/docstubs/stdlib/VERSIONS: _py_abc: 3.7-
.\basedpyright-1.29.5.tar.gz: basedpyright-1.29.5/packages/pyright-internal/typeshed-fallback/stdlib/VERSIONS: _py_abc: 3.7-
.\basedpyright-1.29.5.tar.gz: basedpyright-1.29.5/packages/pyright/dist/typeshed-fallback/stdlib/VERSIONS: _py_abc: 3.7-
.\basedpyright-1.31.7.tar.gz: basedpyright-1.31.7/basedpyright/dist/typeshed-fallback/stdlib/VERSIONS: _py_abc: 3.7-
.\basedpyright-1.31.7.tar.gz: basedpyright-1.31.7/docstubs/stdlib/VERSIONS: _py_abc: 3.7-
.\basedpyright-1.31.7.tar.gz: basedpyright-1.31.7/packages/pyright-internal/typeshed-fallback/stdlib/VERSIONS: _py_abc: 3.7-
.\basedpyright-1.31.7.tar.gz: basedpyright-1.31.7/packages/pyright/dist/typeshed-fallback/stdlib/VERSIONS: _py_abc: 3.7-
.\jedi-0.19.2.tar.gz: jedi-0.19.2/jedi.egg-info/SOURCES.txt: jedi/third_party/typeshed/stdlib/3.7/_py_abc.pyi
.\pipreqs-0.5.0.tar.gz: pipreqs-0.5.0/pipreqs/stdlib: _py_abc
.\pipreqs_fivetran-1.0.1.tar.gz: pipreqs_fivetran-1.0.1/pipreqs/stdlib: _py_abc
.\mypy-1.16.1.tar.gz: mypy-1.16.1/mypy/typeshed/stdlib/VERSIONS: _py_abc: 3.7-
.\mypy-1.16.1.tar.gz: mypy-1.16.1/mypy.egg-info/SOURCES.txt: mypy/typeshed/stdlib/_py_abc.pyi
.\mypy-1.18.2.tar.gz: mypy-1.18.2/mypy/typeshed/stdlib/VERSIONS: _py_abc: 3.7-
.\mypy-1.18.2.tar.gz: mypy-1.18.2/mypy.egg-info/SOURCES.txt: mypy/typeshed/stdlib/_py_abc.pyi
.\pylint-3.3.7.tar.gz: pylint-3.3.7/pylint/checkers/base/basic_error_checker.py: ABC_METACLASSES = {"_py_abc.ABCMeta", "abc.ABCMeta"}  # Python 3.7+,
.\pylint-3.3.7.tar.gz: pylint-3.3.7/pylint/checkers/utils.py: ABC_MODULES = {"abc", "_py_abc"}
.\pylint-3.3.7.tar.gz: pylint-3.3.7/pylint/checkers/variables.py: METACLASS_NAME_TRANSFORMS = {"_py_abc": "abc"}
.\pylint-4.0.1.tar.gz: pylint-4.0.1/pylint/checkers/base/basic_error_checker.py: ABC_METACLASSES = {"_py_abc.ABCMeta", "abc.ABCMeta"}  # Python 3.7+,
.\pylint-4.0.1.tar.gz: pylint-4.0.1/pylint/checkers/utils.py: ABC_MODULES = {"abc", "_py_abc"}
.\pylint-4.0.1.tar.gz: pylint-4.0.1/pylint/checkers/variables.py: METACLASS_NAME_TRANSFORMS = {"_py_abc": "abc"}
.\pytest-archon-0.0.6.tar.gz: pytest-archon-0.0.6/src/pytest_archon/assets/core-module-lists/3.7.txt: _py_abc
.\pytest-archon-0.0.6.tar.gz: pytest-archon-0.0.6/src/pytest_archon/assets/core-module-lists/3.8.txt: _py_abc
.\pytest-archon-0.0.6.tar.gz: pytest-archon-0.0.6/src/pytest_archon/assets/core-module-lists/3.9.txt: _py_abc
.\pytest_archon-0.0.7.tar.gz: pytest_archon-0.0.7/src/pytest_archon/assets/core-module-lists/3.7.txt: _py_abc
.\pytest_archon-0.0.7.tar.gz: pytest_archon-0.0.7/src/pytest_archon/assets/core-module-lists/3.8.txt: _py_abc
.\pytest_archon-0.0.7.tar.gz: pytest_archon-0.0.7/src/pytest_archon/assets/core-module-lists/3.9.txt: _py_abc
.\pyre-check-0.9.23.tar.gz: pyre_check-0.9.23/pyre_check.egg-info/SOURCES.txt: typeshed/stdlib/_py_abc.pyi
.\pyre-check-0.9.25.tar.gz: pyre_check-0.9.25/pyre_check.egg-info/SOURCES.txt: typeshed/stdlib/_py_abc.pyi
.\pyre-check-nightly-0.0.101751800363.tar.gz: pyre_check_nightly-0.0.101751800363/pyre_check_nightly.egg-info/SOURCES.txt: typeshed/stdlib/_py_abc.pyi
.\pyright-1.1.402.tar.gz: pyright-1.1.402/src/pyright/dist/dist/typeshed-fallback/stdlib/VERSIONS: _py_abc: 3.7-
.\pyright-1.1.402.tar.gz: pyright-1.1.402/src/pyright.egg-info/SOURCES.txt: src/pyright/dist/dist/typeshed-fallback/stdlib/_py_abc.pyi
.\pyright-1.1.406.tar.gz: pyright-1.1.406/src/pyright/dist/dist/typeshed-fallback/stdlib/VERSIONS: _py_abc: 3.7-
.\pyright-1.1.406.tar.gz: pyright-1.1.406/src/pyright.egg-info/SOURCES.txt: src/pyright/dist/dist/typeshed-fallback/stdlib/_py_abc.pyi
.\rtest-0.0.36.tar.gz: rtest-0.0.36/ruff/crates/ruff_python_stdlib/src/sys/known_stdlib.rs: | "_py_abc"
.\ruff-0.12.2.tar.gz: ruff-0.12.2/crates/ruff_python_stdlib/src/sys/known_stdlib.rs: | "_py_abc"
.\ruff-0.12.2.tar.gz: ruff-0.12.2/crates/ty_vendored/vendor/typeshed/stdlib/VERSIONS: _py_abc: 3.7-
.\ruff-0.14.1.tar.gz: ruff-0.14.1/crates/ty_vendored/vendor/typeshed/stdlib/VERSIONS: _py_abc: 3.7-
.\ruff-0.14.1.tar.gz: ruff-0.14.1/crates/ruff_python_stdlib/src/sys/known_stdlib.rs: | "_py_abc"
.\typeshed_client-2.7.0.tar.gz: typeshed_client-2.7.0/typeshed_client/typeshed/VERSIONS: _py_abc: 3.7-
.\typeshed_client-2.7.0.tar.gz: typeshed_client-2.7.0/typeshed_client.egg-info/SOURCES.txt: typeshed_client/typeshed/_py_abc.pyi
.\typeshed_client-2.8.2.tar.gz: typeshed_client-2.8.2/typeshed_client/typeshed/VERSIONS: _py_abc: 3.7-
.\typeshed_client-2.8.2.tar.gz: typeshed_client-2.8.2/typeshed_client.egg-info/SOURCES.txt: typeshed_client/typeshed/_py_abc.pyi
.\stdlib_list-0.11.1.tar.gz: stdlib_list-0.11.1/stdlib_list/lists/3.10.txt: _py_abc
.\stdlib_list-0.11.1.tar.gz: stdlib_list-0.11.1/stdlib_list/lists/3.11.txt: _py_abc
.\stdlib_list-0.11.1.tar.gz: stdlib_list-0.11.1/stdlib_list/lists/3.12.txt: _py_abc
.\stdlib_list-0.11.1.tar.gz: stdlib_list-0.11.1/stdlib_list/lists/3.13.txt: _py_abc
.\stdlib_list-0.11.1.tar.gz: stdlib_list-0.11.1/stdlib_list/lists/3.7.txt: _py_abc
.\stdlib_list-0.11.1.tar.gz: stdlib_list-0.11.1/stdlib_list/lists/3.8.txt: _py_abc
.\stdlib_list-0.11.1.tar.gz: stdlib_list-0.11.1/stdlib_list/lists/3.9.txt: _py_abc
.\stdlibs-2025.5.10.tar.gz: stdlibs-2025.5.10/README.md: ['_abc', '_contextvars', '_py_abc', '_queue', '_uuid', '_xxtestfuzz', 'contextvars', 'dataclasses']
.\stdlibs-2025.5.10.tar.gz: stdlibs-2025.5.10/stdlibs/py.py: "_py_abc",
.\stdlibs-2025.5.10.tar.gz: stdlibs-2025.5.10/stdlibs/py3.py: "_py_abc",
.\stdlibs-2025.5.10.tar.gz: stdlibs-2025.5.10/stdlibs/py310.py: "_py_abc",
.\stdlibs-2025.5.10.tar.gz: stdlibs-2025.5.10/stdlibs/py311.py: "_py_abc",
.\stdlibs-2025.5.10.tar.gz: stdlibs-2025.5.10/stdlibs/py312.py: "_py_abc",
.\stdlibs-2025.5.10.tar.gz: stdlibs-2025.5.10/stdlibs/py313.py: "_py_abc",
.\stdlibs-2025.5.10.tar.gz: stdlibs-2025.5.10/stdlibs/py314.py: "_py_abc",
.\stdlibs-2025.5.10.tar.gz: stdlibs-2025.5.10/stdlibs/py37.py: "_py_abc",
.\stdlibs-2025.5.10.tar.gz: stdlibs-2025.5.10/stdlibs/py38.py: "_py_abc",
.\stdlibs-2025.5.10.tar.gz: stdlibs-2025.5.10/stdlibs/py39.py: "_py_abc",
.\stdlibs-2025.5.10.tar.gz: stdlibs-2025.5.10/PKG-INFO: ['_abc', '_contextvars', '_py_abc', '_queue', '_uuid', '_xxtestfuzz', 'contextvars', 'dataclasses']
.\temporalio-1.13.0.tar.gz: temporalio-1.13.0/temporalio/worker/workflow_sandbox/_restrictions.py: "_pickle,_posixshmem,_posixsubprocess,_py_abc,_pydecimal,_pyio,_queue,_random,_scproxy,"
.\temporalio-1.18.1.tar.gz: temporalio-1.18.1/temporalio/worker/workflow_sandbox/_restrictions.py: "_pickle,_posixshmem,_posixsubprocess,_py_abc,_pydecimal,_pyio,_queue,_random,_scproxy,"
.\pytype-2024.10.11.tar.gz: pytype-2024.10.11/pytype/typeshed/stdlib/VERSIONS: _py_abc: 3.7-
.\pytype-2024.10.11.tar.gz: pytype-2024.10.11/pytype.egg-info/SOURCES.txt: pytype/typeshed/stdlib/_py_abc.pyi
.\shed-2025.6.1.tar.gz: shed-2025.6.1/src/shed/_stdlib_module_names/py38.py: "_py_abc",
.\shed-2025.6.1.tar.gz: shed-2025.6.1/src/shed/_stdlib_module_names/py39.py: "_py_abc",
.\ty-0.0.1a13.tar.gz: ty-0.0.1a13/ruff/crates/ty_vendored/vendor/typeshed/stdlib/VERSIONS: _py_abc: 3.7-
.\ty-0.0.1a13.tar.gz: ty-0.0.1a13/ruff/crates/ruff_python_stdlib/src/sys/known_stdlib.rs: | "_py_abc"
.\wrapt-1.17.2.tar.gz: wrapt-1.17.2/tests/test_inheritance_py37.py: import _py_abc
.\wrapt-1.17.2.tar.gz: wrapt-1.17.2/tests/test_inheritance_py37.py: def test_py_abc_inheritance(self):
.\wrapt-1.17.2.tar.gz: wrapt-1.17.2/tests/test_inheritance_py37.py: class A1(metaclass=_py_abc.ABCMeta):
.\wrapt-1.17.3.tar.gz: wrapt-1.17.3/tests/test_inheritance_py37.py: import _py_abc
.\wrapt-1.17.3.tar.gz: wrapt-1.17.3/tests/test_inheritance_py37.py: def test_py_abc_inheritance(self):
.\wrapt-1.17.3.tar.gz: wrapt-1.17.3/tests/test_inheritance_py37.py: class A1(metaclass=_py_abc.ABCMeta):
.\ty-0.0.1a23.tar.gz: ty-0.0.1a23/ruff/crates/ruff_python_stdlib/src/sys/known_stdlib.rs: | "_py_abc"
.\ty-0.0.1a23.tar.gz: ty-0.0.1a23/ruff/crates/ty_vendored/vendor/typeshed/stdlib/VERSIONS: _py_abc: 3.7-

Navigation Menu

Repository navigation

`_py_abc` Python implementation of `abc` is not thread-safe #130095

Bug report

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	def register(cls, subclass):
	"""Register a virtual subclass of an ABC.

	Returns the subclass, to allow usage as a class decorator.
	"""
	if not isinstance(subclass, type):
	raise TypeError("Can only register classes")
	if issubclass(subclass, cls):
	return subclass # Already a subclass
	# Subtle: test for cycles after testing for "already a subclass";
	# this means we allow X.register(X) and interpret it as a no-op.
	if issubclass(cls, subclass):
	# This would create a cycle, which is bad for the algorithm below
	raise RuntimeError("Refusing to create an inheritance cycle")
	cls._abc_registry.add(subclass)
	ABCMeta._abc_invalidation_counter += 1 # Invalidate negative cache
	return subclass

_py_abc Python implementation of abc is not thread-safe #130095

Description

Bug report

Linked PRs

Activity

colesbury commented on Feb 13, 2025

picnixz commented on Feb 14, 2025

colesbury commented on Feb 14, 2025

cfbolz commented on Feb 14, 2025

cfbolz commented on Feb 14, 2025

sergey-miryanov commented on Feb 14, 2025

colesbury commented on Feb 14, 2025

sergey-miryanov commented on Feb 14, 2025

cfbolz commented on Feb 14, 2025

colesbury commented on Feb 14, 2025

sergey-miryanov commented on Oct 20, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`_py_abc` Python implementation of `abc` is not thread-safe #130095