Skip to content

bench 030 claimed it used no node, on the day the node actually arrived - #3286

Merged
Flotapponnier merged 1 commit into
devfrom
fix/hl-frontends-node-source-attribution
Oct 25, 2026
Merged

Flotapponnier merged 1 commit into
devfrom
fix/hl-frontends-node-source-attribution

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

What happened

The Hyperliquid node substitution merged on 2026-10-07 (#2872) and never ran.

The ocb-hyperliquid-frontends image was rebuilt on 2026-10-21 from ~/ocb-build/harness-src/hlf-build on the Paris VPS, a source tree whose Go files are frozen at 2026-10-07 18:09, hours before that merge. It has no nodesource.go and no a.node anywhere in its agg.go. The 21/10 rebuild refreshed only builders.json, which is why Nova appeared on the site (#3130) and the node figures did not. A rebuild of a frozen tree ships no new code, and it looks exactly like a successful deploy.

The absence of hl_frontend_node_days_window_v2 from the container's /metrics was the proof: the gauge is promauto-registered and Set in both branches of if a.node != nil inside publish(), which demonstrably ran. A binary that reaches that line and exports no such gauge does not contain it.

A second trap sat behind the first: rebuilding from the refreshed tree with the Docker cache still produced a binary without the symbols. Only --no-cache compiled the new source. The gate below is what caught it.

What the fix moved

Verified by symbol before the container was touched, then measured through Prometheus either side of the swap:

before after
hl_frontend_volume_usd_7d_v2{builder="fomo"} $263.1M $780.8M (2.97x)
cohort sum(hl_frontend_volume_usd_7d_v2) $1.811B $4.783B (2.64x)
node days in the 30-day window 0 18, plus 10 reconstructions

$780.8M is the figure the node's own seven complete days give, read independently through the reducer before the rebuild, so the bench was publishing roughly a third of the real notional for eighteen days. hl_frontend_ledger_complete returned to 1 three minutes after the swap. The previous image is kept as ocb-hyperliquid-frontends:rollback-20261025.

Why this PR exists

Four statements on the page became false the moment the node arrived, having been accidentally true for as long as the binary was stale: the subtitle, the abstract, the Reproducibility bullet, and the "Where does the data come from" FAQ answer all said no node. That FAQ answer is also rendered as FAQPage JSON-LD, so it was a machine-readable claim.

They now name both sources, state that a whole node day replaces the export's copy of that day outright rather than being averaged with it, and say plainly that the days our node measured are not reproducible from the public feed alone. A new Node substitution methodology bullet carries the mechanism and keeps reconstructions separate from measurements.

The coverage note's arithmetic

The same reading exposed a defect in HlFeedCoverageNote. It took the export's share of the window as windowDays - nodeDays and then described truncatedDaysWindow as a subset of it, which the gauges cannot support: a rebuilt day sits in that remainder, and the truncated count includes days the node replaced. On today's reading that rendered as

the remaining 12 fall back to the export, and 11 of those stop around 12:10

where two days rest on the export. The amber branch was worse: it handed all thirty days to the upstream feed and described the window as reading 43% low, which now understates our own figures.

hlDaySources splits the window into the three sources the page can name, clamped so a scrape landing mid-pass cannot publish a negative day count. The amber branch names the node share, and its label no longer reads "Upstream feed is publishing partial days" over a window with two such days in thirty.

The nodeDays === 0 wording is kept and tested: a node outage, or a rollback to an older image, reproduces exactly the state the page was in until today, and the note has to describe that one correctly too.

Checks

  • pnpm validate 253 specs valid, longest methodology bullet 451 of 500 chars
  • pnpm exec tsc --noEmit clean
  • bun test 778 pass, 0 fail (full suite, not a subset), 4 new cases on hlDaySources
  • no em or en dashes in the spec, no provider set change (spec-provider-set.test.ts green, so no cache key bump needed)

Not in this PR

  • Production reads the same Prometheus and will pick the figures up on its own cache cycle. I have not forced /api/internal/revalidate-aggregate on prod.
  • ~/ocb-build/harness-src/hlf-build is still on the VPS. The new tree is hlf-build-20261025. A durable fix for the frozen-tree trap is a separate question: the sweep below found no other harness affected, but nothing prevents a recurrence.
  • Swept every other build tree on the box against its own harness's last dev commit. rpc-cost and perp-cohort-stats looked stale by date but both publish their 2026-10-22 additions (blockmachine in the catalogue, nova in the metrics), so they were built from a fresher context. hlf-build was the only frozen tree still serving as a build context.

The node substitution merged on 2026-10-07 (#2872) and never ran. The
hyperliquid-frontends image was rebuilt on 2026-10-21 from
~/ocb-build/harness-src/hlf-build, a source tree whose Go files are frozen
at 2026-10-07 18:09, hours before that merge: no nodesource.go, and no
`a.node` anywhere in its agg.go. That rebuild refreshed only builders.json,
which is why Nova appeared on the site and the node figures did not. The
gauges are promauto-registered and Set in both branches of `if a.node != nil`
inside publish(), so their absence from /metrics was the proof the binary had
no node code rather than a node that was declining every day.

Rebuilt today from current dev, verified by symbol before the container was
touched, and the swap moved every figure on the bench:

  FOMO volume 7d      $263.1M -> $780.8M   (2.97x)
  cohort volume 7d    $1.811B -> $4.783B   (2.64x)
  node days / 30            0 -> 18, plus 10 reconstructions

$780.8M is the figure the node's own seven complete days give, read
independently through the reducer, so the bench was publishing roughly a
third of the real notional for eighteen days.

That makes four statements on the page false as of today, where they had
been accidentally true for as long as the binary was stale: the subtitle,
the abstract, the reproducibility bullet and the "where does the data come
from" FAQ answer all said no node. The FAQ answer is also FAQPage JSON-LD,
so it was a machine-readable claim. They now name both sources, say that a
whole node day replaces the export's copy outright rather than being
averaged with it, and say plainly that the days our node measured are not
reproducible from the public feed alone.

The coverage note had a second defect the same reading exposed. It took the
export's share of the window as windowDays - nodeDays and then described
truncatedDaysWindow as a subset of it, which the gauges cannot support: a
rebuilt day sits in that remainder, and the truncated count includes days
the node replaced. On today's reading that rendered as "the remaining 12
fall back to the export, and 11 of those stop around 12:10" where two days
rest on the export. hlDaySources splits the window into the three sources
the page can name, clamped so a scrape landing mid-pass cannot publish a
negative day count, and the amber branch now names the node share instead
of handing all thirty days to the upstream feed.

The nodeDays == 0 wording is kept and tested. A node outage or a rollback to
an older image reproduces exactly the state the page was in until today, and
the note has to describe that one correctly too.
@Flotapponnier
Flotapponnier merged commit 79a0411 into dev Oct 25, 2026
2 checks passed
@Flotapponnier
Flotapponnier deleted the fix/hl-frontends-node-source-attribution branch October 25, 2026 12:56
Flotapponnier added a commit that referenced this pull request Oct 25, 2026
…ed (#3286) (#3289)

The node substitution merged on 2026-10-07 (#2872) and never ran. The
hyperliquid-frontends image was rebuilt on 2026-10-21 from
~/ocb-build/harness-src/hlf-build, a source tree whose Go files are frozen
at 2026-10-07 18:09, hours before that merge: no nodesource.go, and no
`a.node` anywhere in its agg.go. That rebuild refreshed only builders.json,
which is why Nova appeared on the site and the node figures did not. The
gauges are promauto-registered and Set in both branches of `if a.node != nil`
inside publish(), so their absence from /metrics was the proof the binary had
no node code rather than a node that was declining every day.

Rebuilt today from current dev, verified by symbol before the container was
touched, and the swap moved every figure on the bench:

  FOMO volume 7d      $263.1M -> $780.8M   (2.97x)
  cohort volume 7d    $1.811B -> $4.783B   (2.64x)
  node days / 30            0 -> 18, plus 10 reconstructions

$780.8M is the figure the node's own seven complete days give, read
independently through the reducer, so the bench was publishing roughly a
third of the real notional for eighteen days.

That makes four statements on the page false as of today, where they had
been accidentally true for as long as the binary was stale: the subtitle,
the abstract, the reproducibility bullet and the "where does the data come
from" FAQ answer all said no node. The FAQ answer is also FAQPage JSON-LD,
so it was a machine-readable claim. They now name both sources, say that a
whole node day replaces the export's copy outright rather than being
averaged with it, and say plainly that the days our node measured are not
reproducible from the public feed alone.

The coverage note had a second defect the same reading exposed. It took the
export's share of the window as windowDays - nodeDays and then described
truncatedDaysWindow as a subset of it, which the gauges cannot support: a
rebuilt day sits in that remainder, and the truncated count includes days
the node replaced. On today's reading that rendered as "the remaining 12
fall back to the export, and 11 of those stop around 12:10" where two days
rest on the export. hlDaySources splits the window into the three sources
the page can name, clamped so a scrape landing mid-pass cannot publish a
negative day count, and the amber branch now names the node share instead
of handing all thirty days to the upstream feed.

The nodeDays == 0 wording is kept and tested. A node outage or a rollback to
an older image reproduces exactly the state the page was in until today, and
the note has to describe that one correctly too.

(cherry picked from commit 79a0411)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant