Skip to content

the worker kept ranking rows the harness had withdrawn, with no end date - #2950

Merged
Flotapponnier merged 1 commit into
devfrom
fix/stale-carry
Oct 12, 2026
Merged

Flotapponnier merged 1 commit into
devfrom
fix/stale-carry

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

the worker kept ranking rows the harness had withdrawn, with no end date

The carry-forward exists so a transient Prometheus failure does not blank a
row: a provider missing from one sweep keeps its last values and gets a
staleSince stamp. The code says as much, and already refuses to carry a
provider deleted from the spec, "the staleness carry exists for transient
query failures, not for editorial removals".

It had no ceiling, so it also survived a catalogue correction. A row
withdrawn by the harness came back every sweep, forever, and the ledger
never reads meta — so it rendered as availability "live" at a 100% success
rate. schema.ts says dead rows are "never deleted", and they were not.

Bench 283 had three tabs in that state at once, each tracing to a commit:

Solana OnFinality #1 at 12,000,000, last observed 22 h earlier (#2940)
dRPC #2 at 10,500,000, last observed 115 h earlier (#2835), still
carrying the pre-fix 24h-quantile query
Syndica, the real leader at 10,000,000, shown third
Trace Ankr #2 and InstaNodes #3, last observed 60 h earlier (#2930),
both above BlockPI — on a workload their own plans refuse, under
a methodology promising a refusal "shows as its absence from the
trace tab"
Indexer InstaNodes #5, above BlockPI, for an archive mix it declines

Those are the two providers I removed from the catalogue on 2026-10-09 after
reading Ankr's and Instanodes' own pricing pages. The harness obeyed: it
resets every gauge each cycle, and Prometheus returns 10 providers on the
Solana tab, 3 on trace, 8 on indexer. The worker put the rows back.

So the carry now has a ceiling, and the bench's own
expected_freshness_seconds is it — the window the bench already claims its
data is good for, 3 hours on these two. Specs declaring none fall back to a
day: generous, but finite, which "never" was not.

mergeWithPrevious moves to worker/merge.ts to be testable at all. worker/index.ts
refuses to load without KV credentials, which is right for a worker with
nowhere to publish and is also why this merge had no test. Four now cover it:
a short gap still carries, a withdrawal past the window does not, the
fallback ceiling bites, and the existing editorial-removal guard still holds.

Found by the bench audit, which read the live board rather than the catalogue
and so saw what a reader sees.

The carry-forward exists so a transient Prometheus failure does not blank a
row: a provider missing from one sweep keeps its last values and gets a
`staleSince` stamp. The code says as much, and already refuses to carry a
provider deleted from the spec, "the staleness carry exists for transient
query failures, not for editorial removals".

It had no ceiling, so it also survived a catalogue correction. A row
withdrawn by the harness came back every sweep, forever, and the ledger
never reads `meta` — so it rendered as availability "live" at a 100% success
rate. `schema.ts` says dead rows are "never deleted", and they were not.

Bench 283 had three tabs in that state at once, each tracing to a commit:

  Solana   OnFinality #1 at 12,000,000, last observed 22 h earlier (#2940)
           dRPC #2 at 10,500,000, last observed 115 h earlier (#2835), still
           carrying the pre-fix 24h-quantile query
           Syndica, the real leader at 10,000,000, shown third
  Trace    Ankr #2 and InstaNodes #3, last observed 60 h earlier (#2930),
           both above BlockPI — on a workload their own plans refuse, under
           a methodology promising a refusal "shows as its absence from the
           trace tab"
  Indexer  InstaNodes #5, above BlockPI, for an archive mix it declines

Those are the two providers I removed from the catalogue on 2026-10-09 after
reading Ankr's and Instanodes' own pricing pages. The harness obeyed: it
resets every gauge each cycle, and Prometheus returns 10 providers on the
Solana tab, 3 on trace, 8 on indexer. The worker put the rows back.

So the carry now has a ceiling, and the bench's own
`expected_freshness_seconds` is it — the window the bench already claims its
data is good for, 3 hours on these two. Specs declaring none fall back to a
day: generous, but finite, which "never" was not.

mergeWithPrevious moves to worker/merge.ts to be testable at all. worker/index.ts
refuses to load without KV credentials, which is right for a worker with
nowhere to publish and is also why this merge had no test. Four now cover it:
a short gap still carries, a withdrawal past the window does not, the
fallback ceiling bites, and the existing editorial-removal guard still holds.

Found by the bench audit, which read the live board rather than the catalogue
and so saw what a reader sees.
@Flotapponnier
Flotapponnier merged commit 6434640 into dev Oct 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant