Repository navigation
CI: UBSan runs GIL-only and never against the JIT; two configurations already shipped by distros are unsanitized #154927
Description
Feature or enhancement
Proposal
The UBSan job in .github/workflows/build.yml runs against exactly one configuration: GIL-enabled, no JIT. Two configurations that CI already builds — and that distributions already ship — are never sanitized. Both are cheap to add, and I have measured what each would have caught.
1. UBSan × free-threading: true
The sanitizer matrix pairs TSan with free-threading: {false, true} but UBSan only with false:
free-threading:
- false
- true
sanitizer:
- TSan
include:
- check-name: Undefined behavior
sanitizer: UBSan
free-threading: falseSo files that only compile under Py_GIL_DISABLED are never built under UBSan. That is not a hypothetical gap: Python/uniqueid.c:185 shifted a negative per-thread refcount delta left, which is UB, and it fired from interpreter startup and from every GC cycle. Over the full suite with all suppressions disabled:
| UB reports | test failures | failing files | |
|---|---|---|---|
| before | 1762 | 529 | 61 |
| after #154915 | 0 | 0 | 0 |
Once #154915 lands, this configuration is green, so adding it is a regression guard rather than a source of new work.
2. UBSan × --enable-experimental-jit
jit.yml builds the JIT in several configurations and tail-call.yml builds --with-tail-call-interp, but neither is ever combined with a sanitizer. The JIT's own C code — the stencil patcher in Python/jit.c, and the tier-2 optimiser in Python/optimizer*.c which only compiles under -D_Py_TIER2=1 — is therefore never sanitized in CI.
This one has already proven productive when done by hand: gh-139269 (an unaligned uint64_t store in Python/jit.c's patch_* functions, which segfaulted release builds) was found by building --enable-experimental-jit with -fsanitize=address,undefined, and gh-142476 was an ASan-found leak in allocate_executor.
I ran the full test suite against a machine-code JIT build under UBSan, with every entry in Tools/ubsan/suppressions.txt disabled: clean, zero reports, 51,459 tests across 493 files, 4m50s. Verified non-vacuous — _testinternalcapi.get_jit_backend() returns jit and a hot loop produces real machine-code ranges, so this was the copy-and-patch JIT rather than a silent fallback to the uop interpreter.
Since it is already green, this is also a pure regression guard.
Two smaller findings from the same sweep
local-boundsis enabled nowhere.--with-undefined-behavior-sanitizerinjects plain-fsanitize=undefined, andlocal-boundsis not in that group even though it detects genuine UB. I built with it on and ran the full suite: zero findings, including no false positives from the trailing variable-length array idiom, which was the obvious risk. It looks free to add.float-divide-by-zero(also not in the group) yields exactly one hit,Modules/expat/xmlparse.c:869, in vendored libexpat, which deliberately relies on IEEE-754+infand documents that in a comment. Not worth enabling without excluding expat.
On cost
The usual objection to widening the matrix is build time. For reference, a full PGO + ThinLTO + UBSan build — the heaviest combination I tried, including the instrumented build, the profile run, the rebuild and the ThinLTO link — took 9 min 43 s at --jobs=8. A plain UBSan build is far cheaper, and the JIT suite run above took under five minutes.
(That PGO/LTO configuration found nothing new, incidentally. It is worth noting only because --enable-optimizations and --with-lto appear in no workflow at all, while Arch, Debian and python-build-standalone all ship them. UBSan's checks are source-level, so PGO/LTO mostly changes which UB is reached; miscompilation from UB would show up as test failures rather than sanitizer reports, which is a different exercise.)
Linked PRs
- gh-148286: Fix undefined behaviour in per-thread refcount merging #154915 — fixes the free-threaded UB, prerequisite for adding that matrix entry
Has this already been discussed elsewhere?
This is a follow-up to gh-148286.
Activity
StanFromIreland commentedon Jul 30, 2026
How well is this covered by Buildbots? I'm also not sure we can expand it to the JIT currently with the development pause, it does fall under a new feature (albeit an infra one).
matthiasgoergens commentedon Aug 1, 2026
Good questions — I checked the fleet rather than guessing.
Buildbot coverage
Querying
buildbot.python.org/api/v2/builders(1588 builders, allbranches and PR builders included), matching on the
ubsan/asan/nogiltags rather than on builder names:
The UBSan builders are
AMD64 Arch Linux Usan {3.9 … 3.15, 3.x, PR}andAMD64 Arch Linux Usan Function {3.9 … 3.15, 3.x, PR}, all taggedclang, ubsan, sanitizer. They are active, not vestigial: on thedevelopment branch,
AMD64 Arch Linux Usan 3.xandAMD64 Arch Linux Usan Function 3.xboth last built green on 2026-08-08.(The one exception is
AMD64 Arch Linux Usan PR, which last ran 2026-01-05and failed in its
git/cleansteps rather than in any test — its worker isonline and its sibling
Usan Function PRran fine on 2026-07-30, so thislooks like a stale workspace on a builder that only runs when a PR requests
it.)
So buildbots do cover UBSan, on Arch with clang, across every active
branch — and that partly duplicates the GitHub Actions job. I was wrong to
say otherwise.
What is still true, and is what this issue asks for: none of those twenty
builders carries a
nogilorjittag. UBSan × free-threading andUBSan × JIT are covered nowhere — not in GitHub Actions, not on the
buildbots. That matters concretely here, because the
uniqueid.cbug livesin code that only compiles under
Py_GIL_DISABLED, so no existing UBSanbuilder could have caught it, however many of them there are.
Worth adding, since it bears on how much the sanitizer job is worth:
UBSan only makes a selected set of undefined behaviour observable, and
only when you build for it. None of the three bugs behind this proposal
misbehaves in an ordinary build — I checked. So sanitizer coverage is
not redundant with the ordinary Buildbot fleet; it is the only thing
that sees these at all.
The JIT half, and the development pause
Fair, and I do not want to argue release policy. Happy to split this:
existing job, and once gh-148286: Fix undefined behaviour in per-thread refcount merging #154915 lands the configuration is green. On
unpatched main it is worse than red: with this workflow's own
halt_on_error=1, the free-threaded build aborts during interpreterstartup and regrtest never runs. With gh-148286: Fix undefined behaviour in per-thread refcount merging #154915 applied, and using this
workflow's suppression file, options and test command on its own compiler:
Result: SUCCESS, 51,517 tests, noruntime error:reports.(The figure I originally quoted here, "1762 UB reports → 0", did not
survive re-running and has been withdrawn — see the issue body.)
by hand in the meantime. (I originally cited "51,459 tests, zero reports,
with every suppression disabled" as though it were a result on main; it was
measured on a tree that already carried the three fixes under review, so it
shows the JIT adds no UB beyond those. Corrected in the issue body.)
If splitting sounds right I will retitle this issue to the
free-threading half and open a separate one for the JIT.