Skip to content

Bound reference-chain pass rate and root-enum frequency (PROF-16384) - #925

Draft
jbachorik wants to merge 4 commits into
investigate-reference-chainsfrom
reference-chain-pass-pacing
Draft

jbachorik wants to merge 4 commits into
investigate-reference-chainsfrom
reference-chain-pass-pacing

Conversation

@jbachorik

@jbachorik jbachorik commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?:

Bounds the reference-chain collector's safepoint frequency. Two scheduler changes, stacked on #921:

  • Minimum pass spacing (shouldRunPass()): one gate ahead of every authorization lane — GC epoch, canary back-to-back (mult=1), fallback cadence — requires now - _last_pass_ns >= _effective_cadence_ns. A completed GC now makes a pass eligible; it no longer decides when. The pause controller still owns the cadence, so measured safepoint pain feeds back into the rate, and the OOM urgency ramp needs no exemption because it drives the same cadence.
  • Adaptive root-enumeration interval (shouldRunRootEnum(), new): a truncated or errored root enumeration no longer retries on the very next pass. Every attempt waits out max(2s, 20ns per measured in-safepoint cost-ms), capped at 30s. On HotSpot, IterateOverReachableObjects unconditionally runs EscapeBarrier::deoptimize_objects_all_threads() (jvmtiTagMap.cpp:3455), so root-enum frequency is the forced-deoptimization storm's direct lever; the cost-scaled interval makes an expensive walk push the next one out proportionally.

Also adds debug-only *ForTest seams over the scheduling state (release builds keep no-op stubs), removes the unused setReferenceChainSearchStartedForTest0 seam, and extends resetSearchStateForTest() to reset the pacing state.

Motivation:

On prof-analyzer (prod), the collector produced ~900 VM ops in 55s — 575 VM_DeoptimizeFrame + 325 VM_HeapWalkOperation, one safepoint every ~54ms. Two amplifiers: the GC-epoch gate authorized a pass before the cadence check (pass rate tracked the young-GC rate), and a truncated enumeration retried every pass, converting each pass into an unconditional deopt-all enumeration. Staging showed the same shape with zero leak candidates (frontier grown to 382k entries, 0 chains) — #921 stops the candidate-less search; this change bounds the frequency of what legitimately remains. Evidence and staging measurements: PROF-16384.

Additional Notes:

  • Accepted behavior change: the collector's pass rate is now capped by _effective_cadence_ns (10ms–4s, PID-controlled) even when a leak signal is hot; root enumerations run at most once per 2s (more when costly). A truncated enumeration pass does no expansion that pass.
  • Measured (local repro, truncated-enum regime, 90s): pre-fix 793 passes → post-fix 43; deopt-all enumerations 793 → ≤45 (18×). With the new EA-spin repro workers (REFCHAINS_EASPIN=4): DeoptimizeFrame ops 260 → 73 per run, 0 in the no-worker control. Chain emission unchanged (17 = 17).
  • The default-scenario (non-truncated) op volume is expansion-bound and unchanged within run variance — reported as-is on PROF-16384.
  • Agentic review (sphinx, deep): 19 findings, 0 critical/high; 13 fixes applied and re-verified, 6 dismissed with rationale (test-seam policy).

How to test the change?:

  • ReferenceChainSchedulingTest (new): pins both gates in open and closed directions, with positive controls for the negated probes, verified GC-epoch advancement, and the native-constant mirrors; boundary tests cover the root-enum interval edge and the GC-epoch shift clamp.
  • referenceChainsBfsTests.inc / referenceChainsOomTests.inc: canary-lane and refill tests backdate _last_pass_ns for the new gate; new boundary tests.
  • Local repro: REFCHAINS_EASPIN=4 REFCHAINS_ARGS=firstpassbudget=1 utils/run-refchains-repro.sh <out.jfr> 90, then compare DeoptimizeFrame / HeapWalkOperation counts in the safepoint log.
  • Verified green: :ddprof-lib:gtestDebug_referenceChains_ut, :ddprof-test:testDebug -Ptests=ReferenceChainSchedulingTest, full *ReferenceChain* suites, spotlessApply.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-16384

@jbachorik jbachorik added the AI label Oct 30, 2026
@jbachorik
jbachorik force-pushed the investigate-reference-chains branch from 522ee15 to 9f0aeca Compare October 30, 2026 15:13
@dd-octo-sts

dd-octo-sts Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #41086952742 | Commit: 891bcc6 | Duration: 19m 58s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - ✅ - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-10-30 16:29:49 UTC

@jbachorik
jbachorik force-pushed the reference-chain-pass-pacing branch from ba98ff4 to 2ea5836 Compare October 30, 2026 15:36
Add a minimum pass spacing gate ahead of every shouldRunPass authorization
lane (GC epoch, canary back-to-back, fallback cadence), so pass rate tracks
the pause controller's cadence instead of the young-GC rate.

End a pass immediately on a truncated or errored root enumeration instead of
retrying on the next pass, and gate retries on an adaptive interval: 2s floor,
extended by 20ns per millisecond of measured in-safepoint cost, capped at 30s.
IterateOverReachableObjects deoptimizes, on every thread, every compiled frame
whose scope holds a scalar-replaced object, so root-enum frequency is the
deopt-all storm's lever.

Add debug-only *ForTest seams over the scheduling state and remove the unused
setReferenceChainSearchStartedForTest0 seam.
@jbachorik
jbachorik force-pushed the reference-chain-pass-pacing branch from 4f26634 to a6d7e06 Compare October 30, 2026 15:39
@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Pipelines

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: bc30ba4 | Docs | Give us feedback!

Add ReferenceChainSchedulingTest over the spacing and root-enum gates
(open and closed directions, positive controls for the negated probes,
verified GC-epoch advancement). Backdate pass spacing in the canary-lane
and refill gtests for the new gate; add boundary tests for the root-enum
interval edge and the GC-epoch shift clamp. Share assumeDebugBuild() via
ReferenceChainAssertions.
REFCHAINS_EASPIN=<n> starts workers holding scalar-replaced locals live
across a loop back-edge, so root enumerations deoptimize their compiled
frames and the deopt axis of the safepoint storm is reproducible locally.
Document the variable in run-refchains-repro.sh.
@jbachorik
jbachorik force-pushed the reference-chain-pass-pacing branch from a6d7e06 to bc30ba4 Compare October 30, 2026 16:07
@dd-octo-sts

dd-octo-sts Bot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 bc30ba40

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant