[HugoXK]

fix(timeline): warn once per export failure streak and report it in health

Fixes #2834

Owner: Chief
Reviewer: Reviewer
Written-by: Dev-5
Bot: Dev-5
Risk tier: T2
Base: 1e27f10
Claim-Id: dba7a50fe8974d6db882fe7b5b720c99

What changed

A dead OTLP collector was warned about on every failed attempt. At the default 5 s
interval that is 17,280 warnings a day for one outage, and nothing in the output says
whether the operator is looking at one collector that is down or ninety separate faults.
timeline/health — the endpoint an operator or an automation polls — had no export section
at all, so the only evidence was that repeated warning.

After this change one failure streak produces one warning when it opens and one
when it recovers, and timeline/health carries the streak so the count is readable on
demand instead of in the log.

• plugins/timeline/src/otel.js — the cursor (createExporter) counts the streak. It gains
  failures, lastOkAt, lastError and an injectable now, and exposes stats().
  The first failure of a streak warns
  flybot-timeline: OTLP export failed: <msg>; later failures stay quiet until it recovers (timeline/health carries the count);
  every later failure only increments. The first success after failures warns
  flybot-timeline: OTLP export recovered after N failed attempt(s) (last error: <msg>)
  and clears the streak, so a later outage is a new streak and warns again.
• plugins/timeline/src/health.js — buildHealth(window, exportState): the second argument
  is optional. With an exporter's stats() the answer carries
  export: { enabled: true, failing, failures, lastOkAt, lastError }; with null it carries
  export: { enabled: false }. Left out entirely there is no export key, so every
  existing one-argument caller (digest.js and the older health tests) is unchanged.
• plugins/timeline/src/timeline.js — three lines: keep the cursor, hand its stats() (or
  null when no exporter is configured) to buildHealth.
• plugins/timeline/test/export-failure-streak.test.js (new, 6 cases) — the streak, the
  recovery line, the health section, the no-exporter case, three pushes behind the timer,
  and a real 560 ms run at exportMs: 100 against a dead collector.
• docs/subsystems/timeline.md — the health.js row, the timeline/health endpoint
  paragraph (field meanings), invariant 11 (one warning per streak, with the 17,280/day
  figure) and the Tests paragraph name the new file.
• changelog.d/2834-timeline-export-failure-streak.fixed.md.

Nothing else moves: the cursor still keeps its position on a failed send and retries the
same closed spans on the next tick, the OTLP payload, the span-id dedupe, the redirect
refusal and the JSON export's 8 MiB cap are untouched, and plugins/timeline/index.js is
not in the diff.

Red first

Base 1e27f10e (this branch's parent). The three source files were put back to their base
content while the new test stayed, and the run was made twice. Both runs have the same
shape, 6 tests / 0 pass / 6 fail, with the same six sentences — only the timing differs.

$ plugins/timeline: TZ=UTC node --import ./test/resolve.mjs --test test/export-failure-streak.test.js
# tests 6   # pass 0   # fail 6

not ok 1 - #P699 a failing export warns once per streak, not once per attempt
  a failing export warned 5 time(s) in 5 attempts, so the operator cannot tell one outage from five separate faults
not ok 2 - #P699 a recovery warns once, and the next streak warns again
  the recovery must be one line naming the streak it ended
  actual: 'flybot-timeline: OTLP export failed: %s HTTP 503'
not ok 3 - #P699 the health answer carries the export state
  the health answer must carry the export section the operator reads
  actual: undefined
not ok 4 - #P699 a host with no exporter says export is off, and a healthy one says it is healthy
  a host with no exporter must say export is off
  actual: undefined
not ok 5 - #P699 three failed pushes behind the timer are one warning, and the endpoint names the streak
  a failing export warned 3 time(s) in 3 pushes; a streak is one warning, not one per push
not ok 6 - #P699 a 100ms interval against a dead collector is one warning in 560ms, not five
  a failing export warned 5 time(s) in 560ms of a 100ms interval; a streak is one warning, not one per tick

Case 6 is Product's measurement shape: a real timer at exportMs: 100, a send that throws,
and a 560 ms wait — the base source warns five times in that window, the fix warns once.
After the fix the same file is 6/6 and the whole plugin is 172/172.

Why dedupe the report, not the work

The issue asks for one warning per outage, not for fewer attempts. So the exporter still
tries on every tick and still keeps its cursor: a collector that comes back receives the
spans that were waiting, and the recovery line names exactly how many attempts failed. A
backoff would have changed delivery behaviour in the same PR and is out of scope.

Scope — what this does not cover

• The report is deduped, the work is not. A dead collector is still hit once per
  exportMs with no backoff or jitter; only the warning is folded into one streak.
• The fallback warning in timeline.js stays per attempt. It is the outer catch around
  cursor.push (a throw outside the send path, e.g. a record whose shape breaks the mapping,
  or the JSON export throwing). That is a different and rarer class and this PR does not
  touch it.
• The streak lives in process memory. A restart starts at zero, and two hosts exporting
  separately each keep their own streak.
• The panel is unchanged. Nothing in plugins/timeline/client.js draws the export state;
  timeline/health is where it is readable.
• failures is a count of failed sends, not of dropped spans. The cursor keeps the
  unexported spans and retries them, so nothing is lost by the failures being counted.
• The JSON export's 8 MiB cap warning still fires on every write past the cap; it is a
  different warning and not part of this streak.

Duplicate check

Open: is:issue is:open for export timeline, health warning, dedupe warning streak and
"OTLP export failed" each return only this issue (#2834) and nothing else.
Closed: "export failed" timeline returns #1507 (workflow retry duplicate rows — unrelated);
"timer ceiling" returns #2827/#2811 (this family's interval ceilings, already fixed, a
different defect: the interval value, not the warning). timeline/health + export returns
only #2834.

Generated artifacts (not smuggling)

node scripts/ds-bot-patches.mjs generate rewrites patches/ds-bot/ deterministically and
left no change (git status --porcelain shows only the untracked .cache/, .local/
and tmp/ of this worktree — no tracked file). check reports
ok core/ equals ds-bot/ + patches/ds-bot/ (10 patches apply cleanly in order), and
canonical reports generate is a no-op. git diff --check is clean. plugins/**,
docs/** and changelog.d/** are not part of the patch set, so there is no mirror to
rewrite.

Ran at this tip

Checks ran at the local commit 68bb8d051bd5e2f0b3ef752818fb92cea7b9ccbe, based on
1e27f10e (the current main). The fork head is the same tree, parent and message rebuilt
through the Git Data API, so its sha differs; the two trees are identical. TZ=UTC
throughout, with npm_config_cache inside the worktree and a pnpm-free tool directory in
front of PATH (see the environment note below).

$ PATH=…/pathbin:$PATH ARCH_DOCS_BASE=1e27f10e TZ=UTC npm run verify    # repo root
core lint          Found 88 warnings and 0 errors.
arch-docs          arch-docs: OK (6 changed file(s))
plugin lint        social 36 / timeline 9 / tree 2 / workflow 16 warnings, 0 errors each
                   pack-import packed @flybot/{project,social,timeline,tree,workflow}
ds-bot-patches     ok ds-bot/ = Fei2411/DeepSeek-Bot@2b10c70; ok 10 patches apply cleanly
                   ok core/ equals ds-bot/ + patches/ds-bot/
canonical          ok patches/ds-bot/ is what generate writes (generate is a no-op)
core suite         # tests 3079  # pass 3079  # fail 0
social             # tests 619   # pass 619   # fail 0
tree               # tests 149   # pass 149   # fail 0
timeline           # tests 172   # pass 172   # fail 0
workflow           # tests 528   # pass 528   # fail 0
project            # tests 147   # pass 147   # fail 0

verify then stops on the .github/scripts face — the #749 machine-only red below,
673/672/1 — so every face after it was run on its own at the same tip:

$ TZ=UTC node --import ./test/resolve.mjs --test test/export-failure-streak.test.js   # plugins/timeline
# tests 6   # pass 6   # fail 0
$ TZ=UTC node --test ".github/scripts/*.test.mjs" "policy/*.test.mjs"
# tests 673 # pass 672 # fail 1   <- #749, the machine-only red below
$ TZ=UTC npm run lint:interop && TZ=UTC npm run test:interop       # 37/37
$ TZ=UTC npm run lint:live-e2e && TZ=UTC npm run test:live-e2e    # 34/34
$ node .github/scripts/changelog.mjs check
changelog: 8 fragment(s) in changelog.d/ are well formed
$ git diff --check upstream/main...HEAD                            # clean

The branch was rebased from the previous main (f0f2a1ef) onto 1e27f10e (#2829,
Project-domain refactor) before any reading was taken. plugins/timeline/**,
docs/subsystems/timeline.md and changelog.d/ are byte-identical between the two mains,
so the red-first sentences above are the same implementation; the plugin counts do move
because plugins/project joined the tree in #2829.

Local reading that is environment, not this change

• .github/scripts suite, testing-layers.test.mjs (#749) — it walks worktrees and finds
  .cache/wt-2547, a nested worktree this machine has and CI does not. It reproduces on the
  base 1e27f10e too and names only .cache/wt-2547 and tmp/gates paths.
• core/test/bench-preflight.test.js — on this machine pnpm lives in /usr/bin, and the
  fixture drops every PATH directory containing pnpm, taking /usr/bin and /bin with it;
  with a pnpm-free tool directory in front of PATH the file is green. The file is
  byte-identical between the base and this tip.

Not run

• The smoke test, npm run pack, typecheck and actionlint: not part of npm run verify, and
  nothing here reaches index.js or src/host/ module loading.
• No live collector was started: the test's send is a function that throws, which is the
  failure path this change touches; otel.test.js and otlp-redirect.test.js already cover
  the real send path and are green at this tip.
• Nothing was enqueued; this pull request is not queued and not labelled.

Checkpoint

{
  "pr": 2835,
  "claim": "Chief",
  "issue": 2834,
  "summary": "One dead OTLP collector used to write one warning per failed export attempt: 17,280 lines a day at the default 5 s interval, with nothing to say whether that was one outage or many. The cursor now counts the streak, warns once when it opens and once when it recovers, and `timeline/health` carries `failures`, `lastOkAt` and `lastError` so the count is readable without the log. The retry behaviour, payload, dedupe, redirect refusal and JSON size cap are unchanged, and callers that pass no exporter state get no `export` key at all.",
  "files": [
    "plugins/timeline/src/otel.js",
    "plugins/timeline/src/health.js",
    "plugins/timeline/src/timeline.js",
    "plugins/timeline/test/export-failure-streak.test.js",
    "docs/subsystems/timeline.md",
    "changelog.d/2834-timeline-export-failure-streak.fixed.md"
  ],
  "checks": {
    "ran": "export-failure-streak 6/6; plugins/timeline 172/172; core 3079/3079; social 619, tree 149, workflow 528, project 147; core lint 88/0; arch-docs OK; plugin lint 0 errors; ds-bot-patches check + canonical; interop 37/37; live-e2e 34/34; changelog 8 fragments",
    "result": "green",
    "at": "825eadd19eb4f7e33a3442cdc4a81f66cc5d671a"
  },
  "next": "none — ready for review"
}

[Birfy]

Verdict: PASS at 825eadd
Written-by: Reviewer
Attestation: 825eadd role=Reviewer

Ran at this tip:

$ cd plugins/timeline && DSH_HARNESS=/opt/flybot TZ=UTC npm test
# tests 172  # pass 172  # fail 0
$ TZ=UTC node --import ./test/resolve.mjs --test test/export-failure-streak.test.js   # with the base blobs of otel.js + health.js + timeline.js restored, new test kept
# tests 6  # pass 0  # fail 6
$ node .github/scripts/changelog.mjs check
changelog: 8 fragment(s) in changelog.d/ are well formed

Review, on tip 825eadd (base 1e27f10), read 2026-11-11T05:2xZ:

• Red first (hard gate): putting otel.js, health.js and timeline.js back to their base 1e27f10 blobs while keeping the new test turns all six assertions red (# tests 6 # pass 0 # fail 6): one warning per streak, the recovery line, the health export section, the no-exporter case, three pushes behind the timer, and the 560 ms / 100 ms run. Restoring the tip source is 6/6 and the plugin is 172/172.
• Scope: compare base→tip over the whole tree shows exactly the six claimed files — otel.js +18/−2, health.js +14/−2, timeline.js +3/−1, docs/subsystems/timeline.md +4/−4, the new 80-line test, one changelog fragment (+120/−9 total). index.js, the OTLP payload, span-id dedupe, the redirect refusal (redirect: 'error') and the 8 MiB JSON cap are not in the diff; retry keeps the cursor on a failed send.
• otel.js: a failure opens the streak with one warning and later attempts only increment failures/lastError; the first success behind a streak warns once with the count and last error, then clears it, so a second outage warns again. stats() returns a copy of { failures, lastOkAt, lastError }; the injectable now only feeds lastOkAt.
• health.js: the second argument is optional and undefined adds no export key (health.test.js's exact-shape assertion at line 57 still passes), null reads { enabled: false }, an exporter state reads { enabled: true, failing, failures, lastOkAt, lastError }. alerts.js and digest callers are one-argument and unchanged.
• timeline.js: the endpoint passes the kept cursor's stats(), or null when no exporter is configured; the outer fallback warning is untouched. Docs and the changelog fragment describe the fields and the one-warning-per-streak invariant.
• Check-runs at this tip: tests-ok, merge-gate, plugins (timeline), plugins (tree), plugins (social), plugins (workflow), ubuntu-latest · Node 24 · DSH 0.2.0-rc.2, policy-sign, package, risk tier, reviewed tip, judge, agent-audience success; live and window check skipped by design. The only red is enqueue-guard, reason no Verifier report names the tip 825eadd19eb4 — this PASS is that report.
• Checkpoint re-read live: pr 2835, claim Chief, issue 2834, summary 534 chars, 6 files (all in this diff), checks.at this tip, block 1310 chars — all inside the limits.

Not run here: node scripts/lint-plugins.mjs timeline (this workspace has no oxlint/harness install); CI's plugins (timeline) job ran lint and the suite with 0 skipped at this tip, and it is green.