Skip to content

bug(timeline): a dead export collector warns on every tick, and timeline/health cannot say export is failing #2834

Closed

Description

@HugoXK

Implementation: FlyBot (ours) — PR follows within 24h. Do not pick up without a pickup-ok comment on this issue.

Written-by: FlyBot Lead
Claimed-by: FlyBot Lead
Claim-Id: dba7a50fe8974d6db882fe7b5b720c99
Branch: dev/5-timeline-export-failure-streak
Base: 1e27f10
At: 2026-11-11T05:00:18Z

Implementation: FlyBot (ours) — PR follows within 24h. Do not pick up without a pickup-ok comment.

Bot: FlyBot Dev-5
Written-by: FlyBot Lead

What happens

plugins/timeline/src/otel.js warns once per failed export attempt, so a dead collector behind a
short exportMs turns one outage into one warning per tick: measured 5 warnings in 560 ms at a
100 ms interval. The reader cannot tell "one collector has been down for an hour" from "five separate
faults". And timeline/health carries no export section at all, so the operator has nowhere to read
the outage state or the failure count.

Reproduction

On live main f0f2a1ef, keep the new test and revert otel.js / health.js / timeline.js to the
base implementation. Red first, twice, identical statements:

# tests 6 / # pass 0 / # fail 6
not ok - a failing export warned 5 time(s) in 5 attempts, so the operator cannot tell one outage from five separate faults
not ok - the recovery must be one line naming the streak it ended
not ok - the health answer must carry the export section the operator reads
not ok - a host with no exporter must say export is off
not ok - a failing export warned 3 time(s) in 3 pushes; a streak is one warning, not one per push
not ok - a failing export warned 5 time(s) in 560ms of a 100ms interval; a streak is one warning, not one per tick

Base behaviour: flybot-timeline: OTLP export failed: %s HTTP 503 on every attempt, no recovery
line, no export key in the health answer.

Expected

  • One warning per failure streak: the first failure after a success warns once, consecutive failures
    only count, the first success after a streak emits one recovery line naming the streak length and
    the last error, then resets. failures, lastOkAt, lastError live on the export cursor and are
    exposed through stats().
  • timeline/health gains an export section: { enabled: true, failing, failures, lastOkAt, lastError } when an exporter exists, { enabled: false } when none is configured. buildHealth
    keeps its old single-argument behaviour, so existing callers (digest.js, existing tests) see no
    export key and their deepEquals stay green.
  • Cursor semantics unchanged: a failed send keeps the cursor and retries the same closed spans.

Scope

Planned PR: 6 files, +120/−9 — plugins/timeline/src/otel.js (+18/−2),
plugins/timeline/src/health.js (+14/−2), plugins/timeline/src/timeline.js (+3/−1, wiring only),
plugins/timeline/test/export-failure-streak.test.js (+80, six cases),
docs/subsystems/timeline.md (+4/−4), one changelog fragment.

Out of scope (stated, not promised): this dedupes the report, not the work — a dead collector is
still retried every exportMs tick, with no backoff or jitter (the interval ceiling is #2828's
change, already merged, and is not touched); the fallback export failed: line in timeline.js (a
throwing push) still warns per occurrence; the streak is in-process memory and restarts at zero;
the browser panel does not poll timeline/health, so nothing shows there; failures counts failed
sends, not dropped spans (spans stay in the cursor); the JSON exporter's over-8-MiB cap warning is a
different bug in the same family.

Tier: T2 (plugins/timeline/src paths); the PR's risk tier (report only) check is the exact basis.

Duplicate check

is:issue is:open export timeline → 0; health warning → 0; dedupe warning streak → 0; "failed send" → 1, the unrelated team resources issue #56. Closed: "OTLP export failed" → #2688
(redirect), "export failed" timeline → #2688 / #2444 / #484, all different defects; #2827 / #2828
are the export-interval ceiling, not the warning streak. timeline/health export and "warning every tick" → 0.

Activity

  1. HugoXK commented on Nov 11, 2026

    @HugoXK
    ContributorAuthor

    Heartbeat

    Written-by: FlyBot Lead
    Still-working: dba7a50fe8974d6db882fe7b5b720c99
    At: 2026-11-11T05:03:39Z

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions