Repository navigation
the README said 94 benchmarks, there are 253, and eleven of its claims had rotted - #3293
Merged
Merged
Conversation
…s had rotted
Every volatile figure in the README had drifted, the headline one by a factor
of 2.7, and eleven factual claims were wrong when checked against the live
site. A rewrite with fresh hand-typed numbers would reset the clock on the
same failure, so the counts now live in a generated block between markers and
the one number nobody can let rot, the live count, is a badge reading
/api/citable.
What had drifted, measured on dev:
94 live benchmarks 253 spec files, all status: live
50 Go harnesses 70 directories, every one a Go module
32 Q&A pages 49 answers/*.yml
23 alternatives 21 alternatives/*.yml, it went down
8 categories 9 distinct category: values
sweeps every 60 s code default 60, production runs 300
What was false against the live site, each verified with curl before being
rewritten rather than taken from the old text:
- /api/cite/<slug>/bibtex answers 400. The format is bib. ris, apa and txt
also work; bibtex never did.
- /benchmarks/<slug>/opengraph-image 308s to the HTML page and is not an
image. The image is /api/og/<slug>.
- The MCP tool names were all wrong. tools/list returns list_benchmarks,
get_benchmark, search_benchmarks, compare_providers, recommend_provider
and list_answers. query_prom does not exist and never appears.
- /api/openapi.json covers 11 read endpoints, not "every endpoint".
- /api/freshness is cached 30 s, not 5 s.
- The badge is not 360x36, so no dimensions are quoted now.
- Wikidata Q140172649 was deleted on 2026-07-18 for notability. The item is
Q140756438. (public/.well-known/publisher.json still serves the dead id;
that is a separate fix.)
- Renamed benches redirect 308, not 301, and the lowercase 308 normalization
claim has been false since #2343: /benchmarks/Aggregator-Head-Lag answers
404 today. The 410 Gone and compare-canonical claims measure true and stay.
- The journalist citation example quoted Mobula at 0.8 s. Live today is Codex
at 0.24 s. Any frozen example drifts, so the file now shows the shape and
says to quote the headline with its asOf.
- "cold reads in < 100 ms edge" had no measurement behind it. Our own spec
rules forbid exactly that sentence in a benchmark, so it is gone rather
than restated.
- /robots.ts was linked as a route. It is a source file; the route is
/robots.txt.
Three surfaces that exist and were advertised nowhere. pip install
openchainbench, published and working: a clean venv against prod returns 246
benchmarks and the four documented calls succeed. Accept: text/markdown on any
bench, product or hub URL, which is the most useful surface an agent has and
needed no new API. And the IndexNow ping that runs after the deploy smoke test.
Also added: /api/series, /api/citable/<date>, /api/compare, /api/sparkline, the
badge snippet, and /contribute/ai-brief.md.
Deliberately not added: the Hugging Face Space. It is running but serves
2026-10-21 while the dataset is at 2026-10-25, because app.py caches the
snapshot date with lru_cache for the life of the process and the keepalive
workflow stops that process ever restarting. We do not advertise a stale
surface.
The number story is now reconciled rather than asserted. 253 specs, minus the
5 held on staging, minus the 2 retired benches that keep their spec, leaves
246, and the set difference against /api/citable is empty, not merely the same
size. That identity is what the companion PR tests, offline, so the printed
digits and the invariant behind them both have to hold. The sentence says "this
commit publishes" and not "openchainbench.com publishes", because dev and main
have diverged and a README claim should be true of the branch it sits on.
Every URL in the file was fetched before this commit: 35 absolute URLs and 7
local paths, no failures. The architecture section drops its ASCII diagram,
which duplicated docs/architecture.md and so drifted in two places at once,
for a mermaid graph that GitHub renders and that carries no number able to go
stale.
…one about money
openchainbench-e9 reviewed the draft against production and found nine. I
re-measured each one before changing the copy, and refuted one of its
findings: there IS a root data/ on dev, it just holds a single generated
build artifact rather than datasets, so the layout line was wrong for a
different reason than reported.
- /api/citable/<date> is not a history. 2026-06-01, 2026-10-24 and even the
future 2027-01-01 all return the same 384,035 bytes with today's asOf on
every row. The route's own header documents the gap. The row and the
"dated historical snapshot" phrase are gone. My link checker had passed
it because a wrong answer is still a 200.
- Accept: text/markdown covers five routes, not every page: benchmarks,
products, perps, rwa and capital. Measured HTML on /rpc, /hyperliquid and
/chains/<slug>. The five are now named.
- /api/og/<slug> is a real card at a stable URL, but it is not what the
unfurl uses, so it no longer says so.
- "No key, no account, no rate-limit tier" was misleading. Read endpoints
run a 60 per 60 s per-IP bucket and the named AI crawlers are exempt. An
agent author needs that, so it is stated.
- One of the 25 retired benches redirects to a successor instead of
answering 410: polymarket-resolution-delay 308s to pm-resolution-delay.
The block row, the routing bullet and the convention all say both now.
- "Every citable surface carries asOf and dateModified" is true of the JSON
and false of the badge SVG and the share-card PNG. Narrowed to the JSON,
and freshness is documented as epoch milliseconds.
- rankings[].ms holds the raw series value in the benchmark's stored unit,
not milliseconds: bridge-fee puts a percent there. Commented, because
that is a field someone writes code against.
- The identical-conditions rule has documented exceptions (token-scoped
feeds filtered to a pool, dRPC kept in rpc-cost by decision), so it now
says split apart or filtered to a common scope.
- The hand-typed category list named On-ramps and Explorers, which have no
published benchmark, in an order that was wrong for what is published, and
sat outside the generated markers where it would drift again. Deleted.
And the one that matters most. The draft said "no provider pays for
placement" and "nothing here is for sale" next to a repository that carries
ten referral links in src/data/provider-registry.ts, rendered
rel="noopener sponsored", and a funding.json declaring an Optimism Retro
Funding project. That is the sentence a hostile reader screenshots. It is
replaced by a Funding and links section that points at both artifacts, and
by a narrower claim: placement is not for sale and a provider cannot change a
figure, which is true by construction because a referral link is a display
field and every number comes from PromQL in a spec. Who receives a commission
is deliberately not stated here; that is Florent's to decide, not mine.
Also fixed: the preview claim, since staging is a public SSO-free URL and
providers are shown facts before publication by design; video/ described as
the renderer behind the Export video button rather than internal tooling; the
FAQ pointing at the provider-correction issue template; the keyed cohort
measures paid plans, so "public endpoints" became endpoints any customer can
reach; and an undocumented claim that corrections are logged, dropped because
no log exists.
Checks: 35 absolute URLs and 8 local paths fetched again, zero failures. No em
or en dashes. The mermaid block renders: validated by a server-side render
returning an 18 KB SVG with every label present.
…the repo
Florent's ruling on the money claim: nobody funds the project, the outbound
links are affiliate links, and neutrality holds anyway. So the section says
that plainly instead of the "nothing here is for sale" it replaced, and it
carries no typed count of links, because openchainbench-e9 found that count
had already moved today when Gains was added.
It also now discloses what the specs already disclose and the section would
have looked selective for omitting: nine keyed RPC benchmarks state that
Chainstack, GetBlock and QuickNode provided the endpoints they are measured
on, with the plan named. Verified at benchmarks/ethereum-rpc.yml line 29 and
in eight sibling specs. Disclosing it next to the affiliate links is the
difference between a disclosure and a selection.
I was wrong about data/ and e9 was right. git ls-tree does not list it on
dev or on this branch, and git check-ignore points at .gitignore line 44. It
exists in my working tree only because prebuild wrote it. I had read the
filesystem and believed I was reading the tree, which is the same mistake as
trusting a 200 from a link checker: the command answered, but not the
question I asked. The layout no longer mentions it, because a cloner never
sees it.
Four more claims narrowed to what is actually true:
- BibTeX and RIS exports carry a URL and a retrieval date. They carry no
asOf and no licence, so the citable-surface bullet now separates the JSON
surfaces from the exported citations.
- The markdown rewrite rule tests that Accept contains text/markdown and
does NOT contain text/html. It is not about which comes first, so the
sentence says what the rule actually tests.
- The concept DOI resolves to the latest version of the record, which is
v1.0.1 from 2026-06-22, four months old. Saying only "always the latest"
invites someone to cite it for a figure they read today, so the line now
gives the version and sends a current figure to the page and its asOf.
- "Every published figure comes from the PromQL in a benchmark's spec" was
false for the site as a whole: chain TVL, the perp venue context and the
fee comparison are fetched from third-party APIs with no Prometheus in
the path. Narrowed to every benchmark figure, which is the claim the
affiliate argument actually needs.
The stats block is now the one #3292's renderBlock emits, with the two
category rows. The per-category list sums to 246, the same number the
reconciliation sentence derives independently, and #3292 tests that sum.
Checks: 35 absolute URLs and 7 local paths, zero failures. No em or en
dashes. 273 lines.
…the funding paragraph contradicted itself
Three factual defects that four rounds of checking had walked past, all found
by openchainbench-e9 and verified here before changing anything.
LICENSE on dev is a plain MIT text. It never mentions CC-BY-4.0 and never
mentions data, so the line "Dataset CC-BY-4.0, attribution required. See
LICENSE" sent a reader to a file that does not contain the licence it cites.
Inside the repo that licence exists only as the `license` field on every API
response, so that is what the README now points at. A LICENSE-DATA file would
be the stronger artifact and is on Florent's list.
The funding paragraph said "no sponsor and no grant pays for this project" and
then, two sentences later, that the repository declares an Optimism Retro
Funding project. Retro Funding is a grant programme, so the paragraph argued
with itself. funding.json is a registration rather than money received, and it
now says so. The absolute claim is held out of the file until Florent
reconciles it: he told me nobody funds the project, and he told e9 on
2026-10-23 that two measured providers do. Both cannot be true, and it is the
most quotable sentence in the file, so it waits for him rather than for me.
What stays is what is verifiable either way: the affiliate links, the
registration, the keyed cohort's endpoint provision, and the reason none of
them can move a rank.
"Nothing on the page is typed by hand" is false. A spec carries a
hand-written abstract, methodology and FAQ. It is the numbers that are never
typed, so that is what it claims now.
Two sentences still carried the broad PromQL claim that round 4 narrowed in a
third. The site publishes figures with no Prometheus in the path: chain TVL
from api.llama.fi, the perp venue context, the fee comparison. All three now
say benchmark figures. And "metrics a harness in this repository produced"
became "metrics produced by the harnesses in this repository", because I
cannot assert byte-identity between seventy deployed images and this source,
and this session began by finding an image that was eighteen days behind it.
Reader-facing, from e9's fresh-reader pass:
- The first screen said what the project is and how to call it, but never
what question it answers. One sentence now names two, both pointing at
benchmarks that exist.
- The headline paragraph was repeated two paragraphs later. Folded into one.
- A live card sits under the badges, linking to the benchmark it shows, with
a note that GitHub caches it. A stranger now sees what a benchmark looks
like before reading a table.
- "Every JSON figure carries an asOf ... in JSON, Markdown, RSS and an MCP
server" was garbled: a JSON figure cannot be in Markdown. Formats and
fields are separate sentences now.
- "Not a preview service" is a phrase nobody has heard and read as
defensive. It leads with the rule instead: not editable by the measured.
- The routing bullet was five clauses joined by "and". It is three rules.
- "usually a full measurement window and an audit round" overstated two
conditions, neither of which is that pair. "For example" with three.
- "Providers open PRs here" is a claim about history I could not check.
"Providers can open PRs here" is true by policy.
Checks: 36 absolute URLs and 7 local paths, zero failures, including the new
card at /api/og/aggregator-head-lag (200, image/png, 59 KB). No em or en
dashes. 277 lines.
…site shows The citable-surface bullet said every figure is available as JSON, Markdown, RSS and MCP. True of benchmark figures; the site's non-benchmark numbers, the chain TVL and the perp venue context, are on none of those. Same narrowing already applied to the two PromQL sentences and the funding argument, so the file is now consistent about which figures it is making claims for. Last item from openchainbench-e9's final re-run, which otherwise found no false claim: 56 URLs all answering, 9 local paths present, the block test 8 pass, readme:sync --check current, no em or en dashes.
Florent confirmed it on both channels, to me and to openchainbench-e9 when it put the reconciliation question to him: no funding. That settles the conflict round 5 held the sentence for, where his instruction to me said nobody funds the project and e9's note of 2026-10-23 said two measured providers did. The later statement supersedes it and the two channels now agree, so the absolute claim goes in. funding.json is described as a registration and not a payment received, which is entailed by "no grant funds this project" rather than inferred beside it. That was the internal contradiction in the paragraph: a sentence denying grants sitting above a declared grant programme. Everything else in the section stays as it was verified: the affiliate links read in one place only, the keyed cohort's endpoint provision disclosed in nine specs, and the argument that none of it reaches a measurement.
Florent's call: say nothing about funding in the README, no related sentence. So the Funding and links section is gone, and with it the affiliate-link disclosure, the funding.json mention and the keyed endpoint-provision note. Two sentences elsewhere pointed at it and are rewritten rather than left dangling. The scope bullet no longer says we do not sell placement and no longer refers the reader to a section below; it is now just "not a token, a fund, or an RPC vendor", because the claim that nobody outside the method can change a number already stands on its own in the next bullet and was saying the same thing twice. Nothing measured changes, and nothing is left half-said: a grep for sponsor, affiliate, commission, grant, investor, referralUrl, funding.json and "for sale" returns nothing. The keyed cohort's endpoint provision stays disclosed where it already was, in the methodology of each of the nine keyed specs. Checks: 34 absolute URLs and 7 local paths, zero failures. No em or en dashes. 269 lines.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rewrite of README.md. Paired with #3292, which adds the generated-block script and its test and must merge after this one.
Why
Every volatile figure had drifted, the headline one by 2.7x, and eleven factual claims were wrong against the live site. Rewriting with fresh hand-typed numbers would reset the clock on the same failure, so the counts now sit in a generated block between markers, and the one number nobody can let rot, the live count, is a badge reading
/api/citable.status: liveanswers/*.ymlcategory:valuesEleven false claims, each verified with curl before rewriting
/api/cite/<slug>/bibtexanswers 400 and the format isbib./benchmarks/<slug>/opengraph-image308s to HTML and is not an image, the image is/api/og/<slug>. The MCP tool names were all wrong andquery_promdoes not exist./api/openapi.jsoncovers 11 read endpoints, not every endpoint./api/freshnessis cached 30 s, not 5 s. The badge is not 360x36. Wikidata Q140172649 was deleted on 2026-07-18 for notability and the item is Q140756438. Renamed benches redirect 308 not 301, and the lowercase normalization claim has been false since #2343. The journalist citation example quoted a figure 3.3x off today's. "cold reads in < 100 ms edge" had no measurement behind it, which our own spec rules forbid in a bench./robots.tswas linked as a route.Three surfaces that exist and were advertised nowhere
pip install openchainbenchis published and working, verified in a clean venv against production.Accept: text/markdownon any bench, product or hub URL, which is the most agent-useful surface we have and needed no new API. The IndexNow ping after the deploy smoke test. Also added/api/series,/api/citable/<date>,/api/compare,/api/sparkline, the badge snippet, and/contribute/ai-brief.md.Deliberately not added: the Hugging Face Space. It runs but serves 2026-10-21 against a 2026-10-25 dataset, because
app.pycaches the snapshot date withlru_cachefor the life of the process and the keepalive workflow stops that process restarting. We do not advertise a stale surface.The number story reconciles instead of asserting
253 specs, minus the 5 held on staging, minus the 2 retired benches that keep their spec, leaves 246, and the set difference against
/api/citableis empty, not merely the same size. #3292 tests that identity offline. The sentence says "this commit publishes" rather than naming the live site, because dev and main have diverged and a README claim should be true of the branch it sits on; the badge covers production, and a disagreement between the two now tells you a release is pending instead of looking like a contradiction.Checks
docs/architecture.mdand so drifted in two places at once.Built with openchainbench-e9, which ran the live surface audit (65 URLs), the GEO evidence pass and the drift-guard design, and found most of the eleven.