Skip to content

Quarantine flaky reference-chain slow e2e tests (PROF-16305) - #888

Merged
jbachorik merged 1 commit into
mainfrom
prof-16305-quarantine
Oct 23, 2026
Merged

jbachorik merged 1 commit into
mainfrom
prof-16305-quarantine

Conversation

@jbachorik

@jbachorik jbachorik commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?:
Adds four quarantine entries for the reference-chain slow e2e tests against PROF-16305, scoped to *-slow cells. The failing tests keep running and reporting; only the nightly gating is suspended.

Motivation:
Nightly Sanitized Run has failed every night since 2026-10-20 (PR #800): ReferenceChainTrackingTest.shouldReconstructReferrerChainThroughUnboundedCacheLeak fails ~60% of eligible HotSpot 11+ debug/asan slow jobs (20 of ~32 in run 39893765385), plus intermittent failures of shouldReconstructReferrerChainToGcRoot (musl-aarch64 11), ThreadLocalLeakReferenceChainTest.shouldCorrelateThreadLocalHeldLeakChain (3 jobs), and LeakTagCorrelationReferenceChainTest.shouldCorrelateLeakChainTargetTagWithLiveObjectLeakTag (1 job). The nightly signal is unusable until the tests are fixed; this restores it. Fix in progress under PROF-16305 — that PR removes these lines once verified.

Additional Notes:
Intended to be merged before the fix PR, and superseded by it (its last act is deleting these entries).

How to test the change?:
python3 .github/scripts/quarantine.py validate and .github/scripts/tests/test_quarantine.sh both pass with the entries present.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-16305

The reference-chain slow e2e family fails intermittently in the nightly
sanitized run 39893765385: the cache-leak test never observes a
CachedPayload chain[0] (20 of ~32 eligible HotSpot 11+ debug/asan slow
jobs), the two loose child-JVM variants report out-of-pool correlation
tags, and the gc-root variant never observes a ChainLink chain[0] on
musl-aarch64 JDK 11. Quarantine them against *-slow only, so failures
in the regular suite still gate. The entries go away with the PROF-16305
fix PR (#800) once the tests are made reliable again.
@jbachorik jbachorik added the AI label Oct 23, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #39938620060 | Commit: cd74f47 | Duration: 19m 13s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - ✅ - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-10-23 10:45:41 UTC

@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Pipelines

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a6da6bf | Docs | View more details | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 a6da6bf3

@jbachorik
jbachorik marked this pull request as ready for review October 23, 2026 13:29
@jbachorik
jbachorik requested a review from a team as a code owner October 23, 2026 13:29
@jbachorik
jbachorik merged commit c977af2 into main Oct 23, 2026
199 of 208 checks passed
@jbachorik
jbachorik deleted the prof-16305-quarantine branch October 23, 2026 13:29
@github-actions github-actions Bot added this to the 1.52.0 milestone Oct 23, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T13:31:37.408997Z a6da6bf Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant