Repository navigation
read whole Hyperliquid days off our own node instead of half ones off the CDN - #2872
Merged
Merged
Conversation
… the CDN The public per-builder export stops at roughly 12:10 UTC on most days since 8 September. Measured on fomo for 6 October: the export carries $53.1M of notional and $25,288 of builder fees, and the node's own fill stream for the same day carries $192.5M and $93,131. CoinMarketMan, reading a complete feed, published $93.1K for that day. The node agrees with them to within 4%. So the gap was never a question of method or vendor. We were reading the wrong file, and the node that has the right one has been running on the Singapore box the whole time with --write-fills, holding 129 days of hourly fills where every record carries the builder address, the builder fee, the trader, the coin, crossed and closedPnl. Everything the site reports. Four gigabytes a day, which is why nobody had looked. harnesses/hl-fills-reduce reduces a day to about 1.6 MB in 26 seconds and serves it. The consumer uses a day from here INSTEAD of the export, never averaged with it: two readings of one quantity that differ by a factor of two are not two readings. A day the node only half covers is declined outright rather than served as a smaller number, because a fraction of a day that looks whole is the exact bug this is fixing. hl_frontend_node_days_window_v2 publishes how much of the 30-day window came from the node, so the page can say whether it is reading whole days. The schema number is load-bearing on both ends. A day written under an older shape decodes into zeros field by field in Go, silently, so the reducer re-reduces what it does not recognise and the consumer declines it. Also the disk, because the node is now load-bearing and it was a day and a half from full. hl-cleanup.sh on that box had three defects in its ceiling phase: `find | sort | head -1` returns non-zero under pipefail, which with set -e killed the run, and the phase only executes above the ceiling, so the script worked for months and began failing every 15 minutes the hour it was needed; it moved files to a trash directory on the same filesystem, which frees nothing; and it re-walked a 3 TB tree once per deleted file. The two largest directories had no retention at all. Fixed and given retention, the disk went 86% to 49%, and the box is now scraped and alerted on at all, which it was not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Oct 7, 2026
Flotapponnier
added a commit
that referenced
this pull request
Oct 7, 2026
* read whole Hyperliquid days off our own node instead of half ones off the CDN (#2872) The public per-builder export stops at roughly 12:10 UTC on most days since 8 September. Measured on fomo for 6 October: the export carries $53.1M of notional and $25,288 of builder fees, and the node's own fill stream for the same day carries $192.5M and $93,131. CoinMarketMan, reading a complete feed, published $93.1K for that day. The node agrees with them to within 4%. So the gap was never a question of method or vendor. We were reading the wrong file, and the node that has the right one has been running on the Singapore box the whole time with --write-fills, holding 129 days of hourly fills where every record carries the builder address, the builder fee, the trader, the coin, crossed and closedPnl. Everything the site reports. Four gigabytes a day, which is why nobody had looked. harnesses/hl-fills-reduce reduces a day to about 1.6 MB in 26 seconds and serves it. The consumer uses a day from here INSTEAD of the export, never averaged with it: two readings of one quantity that differ by a factor of two are not two readings. A day the node only half covers is declined outright rather than served as a smaller number, because a fraction of a day that looks whole is the exact bug this is fixing. hl_frontend_node_days_window_v2 publishes how much of the 30-day window came from the node, so the page can say whether it is reading whole days. The schema number is load-bearing on both ends. A day written under an older shape decodes into zeros field by field in Go, silently, so the reducer re-reduces what it does not recognise and the consumer declines it. Also the disk, because the node is now load-bearing and it was a day and a half from full. hl-cleanup.sh on that box had three defects in its ceiling phase: `find | sort | head -1` returns non-zero under pipefail, which with set -e killed the run, and the phase only executes above the ceiling, so the script worked for months and began failing every 15 minutes the hour it was needed; it moved files to a trash directory on the same filesystem, which frees nothing; and it re-walked a 3 TB tree once per deleted file. The two largest directories had no retention at all. Fixed and given retention, the disk went 86% to 49%, and the box is now scraped and alerted on at all, which it was not. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * the window is mostly whole days now, so the page should stop shouting (#2873) 26 of the last 30 days are read from our own Hyperliquid node rather than the public export, so fomo's figures went from 0.53x of CoinMarketMan's to 0.89x on volume, 0.88x on revenue, 0.98x on users and 0.96x on the feed day. The disclosure note still carried the amber banner it wore when the window was half missing. It now has two voices. A window built from the export stays a warning, because its totals read near half of a full-feed tracker's. A window that is almost all node days is a footnote naming the few days the node was down, which cost a few per cent. hl_frontend_node_days_window_v2 is what tells them apart, so the page says where its days came from rather than inferring it from the truncation count. Also two things found while checking what the reader actually sees: prom.scalar rounded every value to six significant digits, which is right for a latency and destroys a unix second. 1791244800 is midnight UTC on 2026-10-06 and rounds to 1791240000, which is 22:40 on the 5th, so the "Biggest complete day" card read 2026-10-05 for a day that ended on the 6th. The rounding exists to trim float tails before caching and an integer has no tail, so integers now pass through exact. The previous value had rounded by 20 minutes and stayed inside its own date, which is why this sat unnoticed. And the note read "The remaining 4fall back to": a space between an expression and the text after it is trimmed by the JSX transform. Those sentences are assembled as strings now. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier
added a commit
that referenced
this pull request
Oct 25, 2026
…ed (#3286) The node substitution merged on 2026-10-07 (#2872) and never ran. The hyperliquid-frontends image was rebuilt on 2026-10-21 from ~/ocb-build/harness-src/hlf-build, a source tree whose Go files are frozen at 2026-10-07 18:09, hours before that merge: no nodesource.go, and no `a.node` anywhere in its agg.go. That rebuild refreshed only builders.json, which is why Nova appeared on the site and the node figures did not. The gauges are promauto-registered and Set in both branches of `if a.node != nil` inside publish(), so their absence from /metrics was the proof the binary had no node code rather than a node that was declining every day. Rebuilt today from current dev, verified by symbol before the container was touched, and the swap moved every figure on the bench: FOMO volume 7d $263.1M -> $780.8M (2.97x) cohort volume 7d $1.811B -> $4.783B (2.64x) node days / 30 0 -> 18, plus 10 reconstructions $780.8M is the figure the node's own seven complete days give, read independently through the reducer, so the bench was publishing roughly a third of the real notional for eighteen days. That makes four statements on the page false as of today, where they had been accidentally true for as long as the binary was stale: the subtitle, the abstract, the reproducibility bullet and the "where does the data come from" FAQ answer all said no node. The FAQ answer is also FAQPage JSON-LD, so it was a machine-readable claim. They now name both sources, say that a whole node day replaces the export's copy outright rather than being averaged with it, and say plainly that the days our node measured are not reproducible from the public feed alone. The coverage note had a second defect the same reading exposed. It took the export's share of the window as windowDays - nodeDays and then described truncatedDaysWindow as a subset of it, which the gauges cannot support: a rebuilt day sits in that remainder, and the truncated count includes days the node replaced. On today's reading that rendered as "the remaining 12 fall back to the export, and 11 of those stop around 12:10" where two days rest on the export. hlDaySources splits the window into the three sources the page can name, clamped so a scrape landing mid-pass cannot publish a negative day count, and the amber branch now names the node share instead of handing all thirty days to the upstream feed. The nodeDays == 0 wording is kept and tested. A node outage or a rollback to an older image reproduces exactly the state the page was in until today, and the note has to describe that one correctly too.
Flotapponnier
added a commit
that referenced
this pull request
Oct 25, 2026
…ed (#3286) (#3289) The node substitution merged on 2026-10-07 (#2872) and never ran. The hyperliquid-frontends image was rebuilt on 2026-10-21 from ~/ocb-build/harness-src/hlf-build, a source tree whose Go files are frozen at 2026-10-07 18:09, hours before that merge: no nodesource.go, and no `a.node` anywhere in its agg.go. That rebuild refreshed only builders.json, which is why Nova appeared on the site and the node figures did not. The gauges are promauto-registered and Set in both branches of `if a.node != nil` inside publish(), so their absence from /metrics was the proof the binary had no node code rather than a node that was declining every day. Rebuilt today from current dev, verified by symbol before the container was touched, and the swap moved every figure on the bench: FOMO volume 7d $263.1M -> $780.8M (2.97x) cohort volume 7d $1.811B -> $4.783B (2.64x) node days / 30 0 -> 18, plus 10 reconstructions $780.8M is the figure the node's own seven complete days give, read independently through the reducer, so the bench was publishing roughly a third of the real notional for eighteen days. That makes four statements on the page false as of today, where they had been accidentally true for as long as the binary was stale: the subtitle, the abstract, the reproducibility bullet and the "where does the data come from" FAQ answer all said no node. The FAQ answer is also FAQPage JSON-LD, so it was a machine-readable claim. They now name both sources, say that a whole node day replaces the export's copy outright rather than being averaged with it, and say plainly that the days our node measured are not reproducible from the public feed alone. The coverage note had a second defect the same reading exposed. It took the export's share of the window as windowDays - nodeDays and then described truncatedDaysWindow as a subset of it, which the gauges cannot support: a rebuilt day sits in that remainder, and the truncated count includes days the node replaced. On today's reading that rendered as "the remaining 12 fall back to the export, and 11 of those stop around 12:10" where two days rest on the export. hlDaySources splits the window into the three sources the page can name, clamped so a scrape landing mid-pass cannot publish a negative day count, and the amber branch now names the node share instead of handing all thirty days to the upstream feed. The nodeDays == 0 wording is kept and tested. A node outage or a rollback to an older image reproduces exactly the state the page was in until today, and the note has to describe that one correctly too. (cherry picked from commit 79a0411)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2864 and #2866, which disclosed that our Hyperliquid figures were built from half days. This stops them being half days.
The finding
The public per-builder export stops at roughly 12:10 UTC on most days since 8 September. fomo, 6 October 2026:
The node agrees with CoinMarketMan to within 4%. Validated by reducing hours 00-11 alone and getting $52.25M / $25,129 against the export's $53.13M / $25,288 — the same twelve hours, so the parsing is right and the other twelve were simply missing.
The gap was never a question of method or vendor. We were reading the wrong file. The node that has the right one has been running on the Singapore box the whole time with
--write-fills, holding 129 days of hourly fills where every record carriesbuilder,builderFee, the trader,coin,crossedandclosedPnl. Everything the site reports. Four gigabytes a day, which is why nobody had looked inside.harnesses/hl-fills-reduceReduces a day to ~1.6 MB in 26 seconds and serves it over
:8090(Caddy basic auth, ufw open to the OCB VPS only). Runs on the node box because it reads that machine's local disk.The consumer uses a day from here instead of the export, never averaged with it: two readings of one quantity that differ by a factor of two are not two readings. A day the node only half covers is declined outright rather than served as a smaller number — a fraction of a day that looks whole is the exact bug being fixed.
hl_frontend_node_days_window_v2publishes how much of the 30-day window came from the node, so the page can say whether it is reading whole days.The schema number is load-bearing on both ends: a day written under an older shape decodes into zeros field by field in Go, silently, so the reducer re-reduces what it does not recognise and the consumer declines it.
The disk, because the node is now load-bearing
That box was a day and a half from full and nobody could have known: no Prometheus scraped it.
hl-cleanup.shhad three defects, all in its ceiling phase:find | sort | head -1returns non-zero underpipefail; withset -ethat killed the run. The phase only executes above the ceiling, so the script worked for months and started failing every 15 minutes the hour it was needed. 80% → 86% in 15 hours, unattended.dfnever moved and nothing was freed.And the two largest directories had no retention at all:
replica_cmds(1.1 TB over 85 run-directories, of which hl-node holds exactly one open) andnode_fills_by_block(505 GB over 129 days).Fixed, with retention for both. Disk went 86% → 49%, 1.8 TB free, node uninterrupted throughout. The box is now scraped (
job: hl-fills-reduce) and has six alerts covering disk, node liveness read from the data rather than from systemd, reducer staleness, and window thinness.Checks
go test ./...in both harnesses; the node source has tests for the partial-day refusal, the schema refusal, case-insensitive address matching and secondary-address mergingup{job="hl-fills-reduce"}is 1Backfill of the 42-day window is running on the box. The harness is not yet pointed at the endpoint; that is the deploy step after this merges.
🤖 Generated with Claude Code