Repository navigation
bench 030 claimed it used no node, on the day the node actually arrived - #3286
Merged
Merged
Conversation
The node substitution merged on 2026-10-07 (#2872) and never ran. The hyperliquid-frontends image was rebuilt on 2026-10-21 from ~/ocb-build/harness-src/hlf-build, a source tree whose Go files are frozen at 2026-10-07 18:09, hours before that merge: no nodesource.go, and no `a.node` anywhere in its agg.go. That rebuild refreshed only builders.json, which is why Nova appeared on the site and the node figures did not. The gauges are promauto-registered and Set in both branches of `if a.node != nil` inside publish(), so their absence from /metrics was the proof the binary had no node code rather than a node that was declining every day. Rebuilt today from current dev, verified by symbol before the container was touched, and the swap moved every figure on the bench: FOMO volume 7d $263.1M -> $780.8M (2.97x) cohort volume 7d $1.811B -> $4.783B (2.64x) node days / 30 0 -> 18, plus 10 reconstructions $780.8M is the figure the node's own seven complete days give, read independently through the reducer, so the bench was publishing roughly a third of the real notional for eighteen days. That makes four statements on the page false as of today, where they had been accidentally true for as long as the binary was stale: the subtitle, the abstract, the reproducibility bullet and the "where does the data come from" FAQ answer all said no node. The FAQ answer is also FAQPage JSON-LD, so it was a machine-readable claim. They now name both sources, say that a whole node day replaces the export's copy outright rather than being averaged with it, and say plainly that the days our node measured are not reproducible from the public feed alone. The coverage note had a second defect the same reading exposed. It took the export's share of the window as windowDays - nodeDays and then described truncatedDaysWindow as a subset of it, which the gauges cannot support: a rebuilt day sits in that remainder, and the truncated count includes days the node replaced. On today's reading that rendered as "the remaining 12 fall back to the export, and 11 of those stop around 12:10" where two days rest on the export. hlDaySources splits the window into the three sources the page can name, clamped so a scrape landing mid-pass cannot publish a negative day count, and the amber branch now names the node share instead of handing all thirty days to the upstream feed. The nodeDays == 0 wording is kept and tested. A node outage or a rollback to an older image reproduces exactly the state the page was in until today, and the note has to describe that one correctly too.
Flotapponnier
added a commit
that referenced
this pull request
Oct 25, 2026
…ed (#3286) (#3289) The node substitution merged on 2026-10-07 (#2872) and never ran. The hyperliquid-frontends image was rebuilt on 2026-10-21 from ~/ocb-build/harness-src/hlf-build, a source tree whose Go files are frozen at 2026-10-07 18:09, hours before that merge: no nodesource.go, and no `a.node` anywhere in its agg.go. That rebuild refreshed only builders.json, which is why Nova appeared on the site and the node figures did not. The gauges are promauto-registered and Set in both branches of `if a.node != nil` inside publish(), so their absence from /metrics was the proof the binary had no node code rather than a node that was declining every day. Rebuilt today from current dev, verified by symbol before the container was touched, and the swap moved every figure on the bench: FOMO volume 7d $263.1M -> $780.8M (2.97x) cohort volume 7d $1.811B -> $4.783B (2.64x) node days / 30 0 -> 18, plus 10 reconstructions $780.8M is the figure the node's own seven complete days give, read independently through the reducer, so the bench was publishing roughly a third of the real notional for eighteen days. That makes four statements on the page false as of today, where they had been accidentally true for as long as the binary was stale: the subtitle, the abstract, the reproducibility bullet and the "where does the data come from" FAQ answer all said no node. The FAQ answer is also FAQPage JSON-LD, so it was a machine-readable claim. They now name both sources, say that a whole node day replaces the export's copy outright rather than being averaged with it, and say plainly that the days our node measured are not reproducible from the public feed alone. The coverage note had a second defect the same reading exposed. It took the export's share of the window as windowDays - nodeDays and then described truncatedDaysWindow as a subset of it, which the gauges cannot support: a rebuilt day sits in that remainder, and the truncated count includes days the node replaced. On today's reading that rendered as "the remaining 12 fall back to the export, and 11 of those stop around 12:10" where two days rest on the export. hlDaySources splits the window into the three sources the page can name, clamped so a scrape landing mid-pass cannot publish a negative day count, and the amber branch now names the node share instead of handing all thirty days to the upstream feed. The nodeDays == 0 wording is kept and tested. A node outage or a rollback to an older image reproduces exactly the state the page was in until today, and the note has to describe that one correctly too. (cherry picked from commit 79a0411)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
The Hyperliquid node substitution merged on 2026-10-07 (#2872) and never ran.
The
ocb-hyperliquid-frontendsimage was rebuilt on 2026-10-21 from~/ocb-build/harness-src/hlf-buildon the Paris VPS, a source tree whose Go files are frozen at 2026-10-07 18:09, hours before that merge. It has nonodesource.goand noa.nodeanywhere in itsagg.go. The 21/10 rebuild refreshed onlybuilders.json, which is why Nova appeared on the site (#3130) and the node figures did not. A rebuild of a frozen tree ships no new code, and it looks exactly like a successful deploy.The absence of
hl_frontend_node_days_window_v2from the container's/metricswas the proof: the gauge ispromauto-registered and Set in both branches ofif a.node != nilinsidepublish(), which demonstrably ran. A binary that reaches that line and exports no such gauge does not contain it.A second trap sat behind the first: rebuilding from the refreshed tree with the Docker cache still produced a binary without the symbols. Only
--no-cachecompiled the new source. The gate below is what caught it.What the fix moved
Verified by symbol before the container was touched, then measured through Prometheus either side of the swap:
hl_frontend_volume_usd_7d_v2{builder="fomo"}sum(hl_frontend_volume_usd_7d_v2)$780.8M is the figure the node's own seven complete days give, read independently through the reducer before the rebuild, so the bench was publishing roughly a third of the real notional for eighteen days.
hl_frontend_ledger_completereturned to 1 three minutes after the swap. The previous image is kept asocb-hyperliquid-frontends:rollback-20261025.Why this PR exists
Four statements on the page became false the moment the node arrived, having been accidentally true for as long as the binary was stale: the subtitle, the abstract, the Reproducibility bullet, and the "Where does the data come from" FAQ answer all said no node. That FAQ answer is also rendered as FAQPage JSON-LD, so it was a machine-readable claim.
They now name both sources, state that a whole node day replaces the export's copy of that day outright rather than being averaged with it, and say plainly that the days our node measured are not reproducible from the public feed alone. A new Node substitution methodology bullet carries the mechanism and keeps reconstructions separate from measurements.
The coverage note's arithmetic
The same reading exposed a defect in
HlFeedCoverageNote. It took the export's share of the window aswindowDays - nodeDaysand then describedtruncatedDaysWindowas a subset of it, which the gauges cannot support: a rebuilt day sits in that remainder, and the truncated count includes days the node replaced. On today's reading that rendered aswhere two days rest on the export. The amber branch was worse: it handed all thirty days to the upstream feed and described the window as reading 43% low, which now understates our own figures.
hlDaySourcessplits the window into the three sources the page can name, clamped so a scrape landing mid-pass cannot publish a negative day count. The amber branch names the node share, and its label no longer reads "Upstream feed is publishing partial days" over a window with two such days in thirty.The
nodeDays === 0wording is kept and tested: a node outage, or a rollback to an older image, reproduces exactly the state the page was in until today, and the note has to describe that one correctly too.Checks
pnpm validate253 specs valid, longest methodology bullet 451 of 500 charspnpm exec tsc --noEmitcleanbun test778 pass, 0 fail (full suite, not a subset), 4 new cases onhlDaySourcesspec-provider-set.test.tsgreen, so no cache key bump needed)Not in this PR
/api/internal/revalidate-aggregateon prod.~/ocb-build/harness-src/hlf-buildis still on the VPS. The new tree ishlf-build-20261025. A durable fix for the frozen-tree trap is a separate question: the sweep below found no other harness affected, but nothing prevents a recurrence.rpc-costandperp-cohort-statslooked stale by date but both publish their 2026-10-22 additions (blockmachine in the catalogue, nova in the metrics), so they were built from a fresher context.hlf-buildwas the only frozen tree still serving as a build context.