Repository navigation
readme: counts generated from the repository, and a test that fails when they drift - #3292
Merged
Merged
Conversation
Flotapponnier
added a commit
that referenced
this pull request
Oct 25, 2026
…the repo
Florent's ruling on the money claim: nobody funds the project, the outbound
links are affiliate links, and neutrality holds anyway. So the section says
that plainly instead of the "nothing here is for sale" it replaced, and it
carries no typed count of links, because openchainbench-e9 found that count
had already moved today when Gains was added.
It also now discloses what the specs already disclose and the section would
have looked selective for omitting: nine keyed RPC benchmarks state that
Chainstack, GetBlock and QuickNode provided the endpoints they are measured
on, with the plan named. Verified at benchmarks/ethereum-rpc.yml line 29 and
in eight sibling specs. Disclosing it next to the affiliate links is the
difference between a disclosure and a selection.
I was wrong about data/ and e9 was right. git ls-tree does not list it on
dev or on this branch, and git check-ignore points at .gitignore line 44. It
exists in my working tree only because prebuild wrote it. I had read the
filesystem and believed I was reading the tree, which is the same mistake as
trusting a 200 from a link checker: the command answered, but not the
question I asked. The layout no longer mentions it, because a cloner never
sees it.
Four more claims narrowed to what is actually true:
- BibTeX and RIS exports carry a URL and a retrieval date. They carry no
asOf and no licence, so the citable-surface bullet now separates the JSON
surfaces from the exported citations.
- The markdown rewrite rule tests that Accept contains text/markdown and
does NOT contain text/html. It is not about which comes first, so the
sentence says what the rule actually tests.
- The concept DOI resolves to the latest version of the record, which is
v1.0.1 from 2026-06-22, four months old. Saying only "always the latest"
invites someone to cite it for a figure they read today, so the line now
gives the version and sends a current figure to the page and its asOf.
- "Every published figure comes from the PromQL in a benchmark's spec" was
false for the site as a whole: chain TVL, the perp venue context and the
fee comparison are fetched from third-party APIs with no Prometheus in
the path. Narrowed to every benchmark figure, which is the claim the
affiliate argument actually needs.
The stats block is now the one #3292's renderBlock emits, with the two
category rows. The per-category list sums to 246, the same number the
reconciliation sentence derives independently, and #3292 tests that sum.
Checks: 35 absolute URLs and 7 local paths, zero failures. No em or en
dashes. 273 lines.
Flotapponnier
marked this pull request as ready for review
October 25, 2026 20:34
Flotapponnier
added a commit
that referenced
this pull request
Oct 25, 2026
…s had rotted (#3293) * the README said 94 benchmarks, there are 253, and eleven of its claims had rotted Every volatile figure in the README had drifted, the headline one by a factor of 2.7, and eleven factual claims were wrong when checked against the live site. A rewrite with fresh hand-typed numbers would reset the clock on the same failure, so the counts now live in a generated block between markers and the one number nobody can let rot, the live count, is a badge reading /api/citable. What had drifted, measured on dev: 94 live benchmarks 253 spec files, all status: live 50 Go harnesses 70 directories, every one a Go module 32 Q&A pages 49 answers/*.yml 23 alternatives 21 alternatives/*.yml, it went down 8 categories 9 distinct category: values sweeps every 60 s code default 60, production runs 300 What was false against the live site, each verified with curl before being rewritten rather than taken from the old text: - /api/cite/<slug>/bibtex answers 400. The format is bib. ris, apa and txt also work; bibtex never did. - /benchmarks/<slug>/opengraph-image 308s to the HTML page and is not an image. The image is /api/og/<slug>. - The MCP tool names were all wrong. tools/list returns list_benchmarks, get_benchmark, search_benchmarks, compare_providers, recommend_provider and list_answers. query_prom does not exist and never appears. - /api/openapi.json covers 11 read endpoints, not "every endpoint". - /api/freshness is cached 30 s, not 5 s. - The badge is not 360x36, so no dimensions are quoted now. - Wikidata Q140172649 was deleted on 2026-07-18 for notability. The item is Q140756438. (public/.well-known/publisher.json still serves the dead id; that is a separate fix.) - Renamed benches redirect 308, not 301, and the lowercase 308 normalization claim has been false since #2343: /benchmarks/Aggregator-Head-Lag answers 404 today. The 410 Gone and compare-canonical claims measure true and stay. - The journalist citation example quoted Mobula at 0.8 s. Live today is Codex at 0.24 s. Any frozen example drifts, so the file now shows the shape and says to quote the headline with its asOf. - "cold reads in < 100 ms edge" had no measurement behind it. Our own spec rules forbid exactly that sentence in a benchmark, so it is gone rather than restated. - /robots.ts was linked as a route. It is a source file; the route is /robots.txt. Three surfaces that exist and were advertised nowhere. pip install openchainbench, published and working: a clean venv against prod returns 246 benchmarks and the four documented calls succeed. Accept: text/markdown on any bench, product or hub URL, which is the most useful surface an agent has and needed no new API. And the IndexNow ping that runs after the deploy smoke test. Also added: /api/series, /api/citable/<date>, /api/compare, /api/sparkline, the badge snippet, and /contribute/ai-brief.md. Deliberately not added: the Hugging Face Space. It is running but serves 2026-10-21 while the dataset is at 2026-10-25, because app.py caches the snapshot date with lru_cache for the life of the process and the keepalive workflow stops that process ever restarting. We do not advertise a stale surface. The number story is now reconciled rather than asserted. 253 specs, minus the 5 held on staging, minus the 2 retired benches that keep their spec, leaves 246, and the set difference against /api/citable is empty, not merely the same size. That identity is what the companion PR tests, offline, so the printed digits and the invariant behind them both have to hold. The sentence says "this commit publishes" and not "openchainbench.com publishes", because dev and main have diverged and a README claim should be true of the branch it sits on. Every URL in the file was fetched before this commit: 35 absolute URLs and 7 local paths, no failures. The architecture section drops its ASCII diagram, which duplicated docs/architecture.md and so drifted in two places at once, for a mermaid graph that GitHub renders and that carries no number able to go stale. * round 3 review: nine claims that were false or unverifiable, and the one about money openchainbench-e9 reviewed the draft against production and found nine. I re-measured each one before changing the copy, and refuted one of its findings: there IS a root data/ on dev, it just holds a single generated build artifact rather than datasets, so the layout line was wrong for a different reason than reported. - /api/citable/<date> is not a history. 2026-06-01, 2026-10-24 and even the future 2027-01-01 all return the same 384,035 bytes with today's asOf on every row. The route's own header documents the gap. The row and the "dated historical snapshot" phrase are gone. My link checker had passed it because a wrong answer is still a 200. - Accept: text/markdown covers five routes, not every page: benchmarks, products, perps, rwa and capital. Measured HTML on /rpc, /hyperliquid and /chains/<slug>. The five are now named. - /api/og/<slug> is a real card at a stable URL, but it is not what the unfurl uses, so it no longer says so. - "No key, no account, no rate-limit tier" was misleading. Read endpoints run a 60 per 60 s per-IP bucket and the named AI crawlers are exempt. An agent author needs that, so it is stated. - One of the 25 retired benches redirects to a successor instead of answering 410: polymarket-resolution-delay 308s to pm-resolution-delay. The block row, the routing bullet and the convention all say both now. - "Every citable surface carries asOf and dateModified" is true of the JSON and false of the badge SVG and the share-card PNG. Narrowed to the JSON, and freshness is documented as epoch milliseconds. - rankings[].ms holds the raw series value in the benchmark's stored unit, not milliseconds: bridge-fee puts a percent there. Commented, because that is a field someone writes code against. - The identical-conditions rule has documented exceptions (token-scoped feeds filtered to a pool, dRPC kept in rpc-cost by decision), so it now says split apart or filtered to a common scope. - The hand-typed category list named On-ramps and Explorers, which have no published benchmark, in an order that was wrong for what is published, and sat outside the generated markers where it would drift again. Deleted. And the one that matters most. The draft said "no provider pays for placement" and "nothing here is for sale" next to a repository that carries ten referral links in src/data/provider-registry.ts, rendered rel="noopener sponsored", and a funding.json declaring an Optimism Retro Funding project. That is the sentence a hostile reader screenshots. It is replaced by a Funding and links section that points at both artifacts, and by a narrower claim: placement is not for sale and a provider cannot change a figure, which is true by construction because a referral link is a display field and every number comes from PromQL in a spec. Who receives a commission is deliberately not stated here; that is Florent's to decide, not mine. Also fixed: the preview claim, since staging is a public SSO-free URL and providers are shown facts before publication by design; video/ described as the renderer behind the Export video button rather than internal tooling; the FAQ pointing at the provider-correction issue template; the keyed cohort measures paid plans, so "public endpoints" became endpoints any customer can reach; and an undocumented claim that corrections are logged, dropped because no log exists. Checks: 35 absolute URLs and 8 local paths fetched again, zero failures. No em or en dashes. The mermaid block renders: validated by a server-side render returning an 18 KB SVG with every label present. * round 4: the funding wording Florent settled, and data/ was never in the repo Florent's ruling on the money claim: nobody funds the project, the outbound links are affiliate links, and neutrality holds anyway. So the section says that plainly instead of the "nothing here is for sale" it replaced, and it carries no typed count of links, because openchainbench-e9 found that count had already moved today when Gains was added. It also now discloses what the specs already disclose and the section would have looked selective for omitting: nine keyed RPC benchmarks state that Chainstack, GetBlock and QuickNode provided the endpoints they are measured on, with the plan named. Verified at benchmarks/ethereum-rpc.yml line 29 and in eight sibling specs. Disclosing it next to the affiliate links is the difference between a disclosure and a selection. I was wrong about data/ and e9 was right. git ls-tree does not list it on dev or on this branch, and git check-ignore points at .gitignore line 44. It exists in my working tree only because prebuild wrote it. I had read the filesystem and believed I was reading the tree, which is the same mistake as trusting a 200 from a link checker: the command answered, but not the question I asked. The layout no longer mentions it, because a cloner never sees it. Four more claims narrowed to what is actually true: - BibTeX and RIS exports carry a URL and a retrieval date. They carry no asOf and no licence, so the citable-surface bullet now separates the JSON surfaces from the exported citations. - The markdown rewrite rule tests that Accept contains text/markdown and does NOT contain text/html. It is not about which comes first, so the sentence says what the rule actually tests. - The concept DOI resolves to the latest version of the record, which is v1.0.1 from 2026-06-22, four months old. Saying only "always the latest" invites someone to cite it for a figure they read today, so the line now gives the version and sends a current figure to the page and its asOf. - "Every published figure comes from the PromQL in a benchmark's spec" was false for the site as a whole: chain TVL, the perp venue context and the fee comparison are fetched from third-party APIs with no Prometheus in the path. Narrowed to every benchmark figure, which is the claim the affiliate argument actually needs. The stats block is now the one #3292's renderBlock emits, with the two category rows. The per-category list sums to 246, the same number the reconciliation sentence derives independently, and #3292 tests that sum. Checks: 35 absolute URLs and 7 local paths, zero failures. No em or en dashes. 273 lines. * round 5: the licence pointed at a file that does not contain it, and the funding paragraph contradicted itself Three factual defects that four rounds of checking had walked past, all found by openchainbench-e9 and verified here before changing anything. LICENSE on dev is a plain MIT text. It never mentions CC-BY-4.0 and never mentions data, so the line "Dataset CC-BY-4.0, attribution required. See LICENSE" sent a reader to a file that does not contain the licence it cites. Inside the repo that licence exists only as the `license` field on every API response, so that is what the README now points at. A LICENSE-DATA file would be the stronger artifact and is on Florent's list. The funding paragraph said "no sponsor and no grant pays for this project" and then, two sentences later, that the repository declares an Optimism Retro Funding project. Retro Funding is a grant programme, so the paragraph argued with itself. funding.json is a registration rather than money received, and it now says so. The absolute claim is held out of the file until Florent reconciles it: he told me nobody funds the project, and he told e9 on 2026-10-23 that two measured providers do. Both cannot be true, and it is the most quotable sentence in the file, so it waits for him rather than for me. What stays is what is verifiable either way: the affiliate links, the registration, the keyed cohort's endpoint provision, and the reason none of them can move a rank. "Nothing on the page is typed by hand" is false. A spec carries a hand-written abstract, methodology and FAQ. It is the numbers that are never typed, so that is what it claims now. Two sentences still carried the broad PromQL claim that round 4 narrowed in a third. The site publishes figures with no Prometheus in the path: chain TVL from api.llama.fi, the perp venue context, the fee comparison. All three now say benchmark figures. And "metrics a harness in this repository produced" became "metrics produced by the harnesses in this repository", because I cannot assert byte-identity between seventy deployed images and this source, and this session began by finding an image that was eighteen days behind it. Reader-facing, from e9's fresh-reader pass: - The first screen said what the project is and how to call it, but never what question it answers. One sentence now names two, both pointing at benchmarks that exist. - The headline paragraph was repeated two paragraphs later. Folded into one. - A live card sits under the badges, linking to the benchmark it shows, with a note that GitHub caches it. A stranger now sees what a benchmark looks like before reading a table. - "Every JSON figure carries an asOf ... in JSON, Markdown, RSS and an MCP server" was garbled: a JSON figure cannot be in Markdown. Formats and fields are separate sentences now. - "Not a preview service" is a phrase nobody has heard and read as defensive. It leads with the rule instead: not editable by the measured. - The routing bullet was five clauses joined by "and". It is three rules. - "usually a full measurement window and an audit round" overstated two conditions, neither of which is that pair. "For example" with three. - "Providers open PRs here" is a claim about history I could not check. "Providers can open PRs here" is true by policy. Checks: 36 absolute URLs and 7 local paths, zero failures, including the new card at /api/og/aggregator-head-lag (200, image/png, 59 KB). No em or en dashes. 277 lines. * narrow the last broad claim: benchmark figures, not every figure the site shows The citable-surface bullet said every figure is available as JSON, Markdown, RSS and MCP. True of benchmark figures; the site's non-benchmark numbers, the chain TVL and the perp venue context, are on none of those. Same narrowing already applied to the two PromQL sentences and the funding argument, so the file is now consistent about which figures it is making claims for. Last item from openchainbench-e9's final re-run, which otherwise found no false claim: 56 URLs all answering, 9 local paths present, the block test 8 pass, readme:sync --check current, no em or en dashes. * state the funding plainly: there is none Florent confirmed it on both channels, to me and to openchainbench-e9 when it put the reconciliation question to him: no funding. That settles the conflict round 5 held the sentence for, where his instruction to me said nobody funds the project and e9's note of 2026-10-23 said two measured providers did. The later statement supersedes it and the two channels now agree, so the absolute claim goes in. funding.json is described as a registration and not a payment received, which is entailed by "no grant funds this project" rather than inferred beside it. That was the internal contradiction in the paragraph: a sentence denying grants sitting above a declared grant programme. Everything else in the section stays as it was verified: the affiliate links read in one place only, the keyed cohort's endpoint provision disclosed in nine specs, and the argument that none of it reaches a measurement. * drop the funding section and every sentence touching it Florent's call: say nothing about funding in the README, no related sentence. So the Funding and links section is gone, and with it the affiliate-link disclosure, the funding.json mention and the keyed endpoint-provision note. Two sentences elsewhere pointed at it and are rewritten rather than left dangling. The scope bullet no longer says we do not sell placement and no longer refers the reader to a section below; it is now just "not a token, a fund, or an RPC vendor", because the claim that nobody outside the method can change a number already stands on its own in the next bullet and was saying the same thing twice. Nothing measured changes, and nothing is left half-said: a grep for sponsor, affiliate, commission, grant, investor, referralUrl, funding.json and "for sale" returns nothing. The keyed cohort's endpoint provision stays disclosed where it already was, in the methodology of each of the nine keyed specs. Checks: 34 absolute URLs and 7 local paths, zero failures. No em or en dashes. 269 lines.
…hen they drift The README said 94 live benchmarks while the repository held 253 specs; every number in it had been typed by hand. `pnpm readme:sync` now regenerates one block between markers from the tree alone (spec files, harness directories, answers, alternatives, categories, the dev-only and retired slug sets), with a reconciliation sentence: specs minus held on staging minus retired-with-spec equals what a release publishes (253 - 5 - 2 = 246, which /api/citable serves today). The block's date is the date of the last count change, not of the last run, so it only moves when a number does. src/lib/readme-stats.test.ts runs offline in `pnpm test`, asserts the block, the identity behind the sentence and that every harness directory is a Go module, and names `pnpm readme:sync` on failure. `--check` gives the same verdict from the command line. Lands after the README rewrite that introduces the markers; until then the two block tests fail by design.
…r (24 and 1 today)
…gory list Derived offline from the specs minus the dev-only and retired sets, so the test stays offline; today it matches /api/citable exactly (7 categories, RPCs 170 ... NFT APIs 1). 'Unchanged' now compares the rendered block with its own date instead of a hand-kept list of numbers, so a new row can never be forgotten by the date rule.
Flotapponnier
force-pushed
the
feat/readme-stats-sync
branch
from
October 25, 2026 21:41
c9998d0 to
a3d4e7e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The README said 94 live benchmarks while the repository held 253 specs;
every number in it had been typed by hand.
pnpm readme:syncnowregenerates one block between markers from the tree alone (spec files,
harness directories, answers, alternatives, categories, the dev-only and
retired slug sets), with a reconciliation sentence: specs minus held on
staging minus retired-with-spec equals what a release publishes (253 - 5 -
2 = 246, which /api/citable serves today).
The block's date is the date of the last count change, not of the last
run, so it only moves when a number does. src/lib/readme-stats.test.ts
runs offline in
pnpm test, asserts the block, the identity behind thesentence and that every harness directory is a Go module, and names
pnpm readme:syncon failure.--checkgives the same verdict from thecommand line.
Lands after the README rewrite that introduces the markers; until then the
two block tests fail by design.