Repository navigation
hotfix(prod): Vercel cost cuts from #2342 (ISR purge throttle, OG cache, slim middleware, maxDuration) - #2343
Merged
Merged
Conversation
…ddleware, bound function time (#2342) Cost drivers found from prod logs and the codebase, and what changes: 1. The materialize worker calls /api/internal/revalidate-aggregate after every 60 s publish, and the hook fired revalidateTag("benchmarks"), which invalidated every ISR page each minute. Every crawler hit was a full render (x-vercel-cache: STALE on every page). The hook is now throttled cluster-wide (Upstash SET NX EX) to one purge per REVALIDATE_MIN_INTERVAL_SEC, default 600; the data caches underneath already expire on their own timers. Bench pages move from a meaningless 3600 to a real 600 s revalidate. 2. 37 of 41 opengraph/twitter image routes had no revalidate and were served with max-age=0, so every share-card or crawler fetch ran satori (1-2 s CPU). All 41 now revalidate every 24 h; the two with explicit cache-control headers are aligned to 86400. 3. Middleware ran on every /benchmarks, /products, /answers, /compare request (about half of all site traffic, billed per edge invocation) to do lowercase 308s, one renamed-bench 301, compare canonicalisation, query stripping on four API routes, and 410s. It now matches only mixed-case paths (regex admitting an uppercase letter; verified in a production build that lowercase live pages never invoke it) plus the literal list of retired URLs, pinned to removed-benches.ts by a test. Renamed benches moved to next.config redirects(); query stripping into the three dynamic API handlers (openapi.json stays ISR); the compare page already canonicalised pairs itself. 4. Page maxDuration 300 -> 120 (they read 66 KB blobs now, not Prom); /api/aggregate upstream timeout 70 s -> 40 s with maxDuration 90. Provisioned memory is billed for the whole wall time. 5. Provider index: the ~2.2 MB unstable_cache value never stored on Vercel ("items over 2MB can not be cached"), so buildProviders ran on every render. It is memoized in-process for 60 s, and the worker now publishes it as providers.json in a normalized wire format (~1 MB, so the data cache holds it) plus a ~100 KB index.json that the bench page reads instead of the 8 MB aggregate for its related rail. Both fall back to the previous path until the worker is redeployed. Measured: the index build itself is 9 ms; the win is the aggregate fetch/parse, not the ranking. Verified: bun tests (259), typecheck, validate; production build + next start: all pages static (x-nextjs-cache HIT), no static-to-dynamic errors, OG routes ISR-cached, case redirects 308, renamed bench 308, hook returns {throttled:true} after the first call. Claude-Session: https://claude.ai/code/session_01L4rjxemTvgRwrCG6D2iK5D Co-authored-by: Flotapponnier <florent@mobula.io> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 2253521)
…nches.ts (perp-pf-ratio is dev-only) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4rjxemTvgRwrCG6D2iK5D
Flotapponnier
added a commit
that referenced
this pull request
Sep 13, 2026
…itive) (#2344) Prod logs after #2343 still showed edge-middleware on lowercase /compare, /products and /benchmarks paths: the [A-Z] custom pattern is case-insensitive on Vercel's router (it was case-sensitive under next start, which is what the branch was verified against). Mixed-case URLs now serve the cached lowercase page with its lowercase canonical instead of a 308; the middleware runs only on the literal list of retired URLs. Claude-Session: https://claude.ai/code/session_01L4rjxemTvgRwrCG6D2iK5D Co-authored-by: Flotapponnier <florent@mobula.io> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier
added a commit
that referenced
this pull request
Sep 13, 2026
…e case-insensitive) (#2345) Prod logs after #2343 still showed edge-middleware on lowercase /compare, /products and /benchmarks paths: the [A-Z] custom pattern is case-insensitive on Vercel's router (it was case-sensitive under next start, which is what the branch was verified against). Mixed-case URLs now serve the cached lowercase page with its lowercase canonical instead of a 308; the middleware runs only on the literal list of retired URLs. Claude-Session: https://claude.ai/code/session_01L4rjxemTvgRwrCG6D2iK5D (cherry picked from commit 69f06d6) Co-authored-by: Flotapponnier <florent@mobula.io> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier
added a commit
that referenced
this pull request
Oct 25, 2026
…s had rotted (#3293) * the README said 94 benchmarks, there are 253, and eleven of its claims had rotted Every volatile figure in the README had drifted, the headline one by a factor of 2.7, and eleven factual claims were wrong when checked against the live site. A rewrite with fresh hand-typed numbers would reset the clock on the same failure, so the counts now live in a generated block between markers and the one number nobody can let rot, the live count, is a badge reading /api/citable. What had drifted, measured on dev: 94 live benchmarks 253 spec files, all status: live 50 Go harnesses 70 directories, every one a Go module 32 Q&A pages 49 answers/*.yml 23 alternatives 21 alternatives/*.yml, it went down 8 categories 9 distinct category: values sweeps every 60 s code default 60, production runs 300 What was false against the live site, each verified with curl before being rewritten rather than taken from the old text: - /api/cite/<slug>/bibtex answers 400. The format is bib. ris, apa and txt also work; bibtex never did. - /benchmarks/<slug>/opengraph-image 308s to the HTML page and is not an image. The image is /api/og/<slug>. - The MCP tool names were all wrong. tools/list returns list_benchmarks, get_benchmark, search_benchmarks, compare_providers, recommend_provider and list_answers. query_prom does not exist and never appears. - /api/openapi.json covers 11 read endpoints, not "every endpoint". - /api/freshness is cached 30 s, not 5 s. - The badge is not 360x36, so no dimensions are quoted now. - Wikidata Q140172649 was deleted on 2026-07-18 for notability. The item is Q140756438. (public/.well-known/publisher.json still serves the dead id; that is a separate fix.) - Renamed benches redirect 308, not 301, and the lowercase 308 normalization claim has been false since #2343: /benchmarks/Aggregator-Head-Lag answers 404 today. The 410 Gone and compare-canonical claims measure true and stay. - The journalist citation example quoted Mobula at 0.8 s. Live today is Codex at 0.24 s. Any frozen example drifts, so the file now shows the shape and says to quote the headline with its asOf. - "cold reads in < 100 ms edge" had no measurement behind it. Our own spec rules forbid exactly that sentence in a benchmark, so it is gone rather than restated. - /robots.ts was linked as a route. It is a source file; the route is /robots.txt. Three surfaces that exist and were advertised nowhere. pip install openchainbench, published and working: a clean venv against prod returns 246 benchmarks and the four documented calls succeed. Accept: text/markdown on any bench, product or hub URL, which is the most useful surface an agent has and needed no new API. And the IndexNow ping that runs after the deploy smoke test. Also added: /api/series, /api/citable/<date>, /api/compare, /api/sparkline, the badge snippet, and /contribute/ai-brief.md. Deliberately not added: the Hugging Face Space. It is running but serves 2026-10-21 while the dataset is at 2026-10-25, because app.py caches the snapshot date with lru_cache for the life of the process and the keepalive workflow stops that process ever restarting. We do not advertise a stale surface. The number story is now reconciled rather than asserted. 253 specs, minus the 5 held on staging, minus the 2 retired benches that keep their spec, leaves 246, and the set difference against /api/citable is empty, not merely the same size. That identity is what the companion PR tests, offline, so the printed digits and the invariant behind them both have to hold. The sentence says "this commit publishes" and not "openchainbench.com publishes", because dev and main have diverged and a README claim should be true of the branch it sits on. Every URL in the file was fetched before this commit: 35 absolute URLs and 7 local paths, no failures. The architecture section drops its ASCII diagram, which duplicated docs/architecture.md and so drifted in two places at once, for a mermaid graph that GitHub renders and that carries no number able to go stale. * round 3 review: nine claims that were false or unverifiable, and the one about money openchainbench-e9 reviewed the draft against production and found nine. I re-measured each one before changing the copy, and refuted one of its findings: there IS a root data/ on dev, it just holds a single generated build artifact rather than datasets, so the layout line was wrong for a different reason than reported. - /api/citable/<date> is not a history. 2026-06-01, 2026-10-24 and even the future 2027-01-01 all return the same 384,035 bytes with today's asOf on every row. The route's own header documents the gap. The row and the "dated historical snapshot" phrase are gone. My link checker had passed it because a wrong answer is still a 200. - Accept: text/markdown covers five routes, not every page: benchmarks, products, perps, rwa and capital. Measured HTML on /rpc, /hyperliquid and /chains/<slug>. The five are now named. - /api/og/<slug> is a real card at a stable URL, but it is not what the unfurl uses, so it no longer says so. - "No key, no account, no rate-limit tier" was misleading. Read endpoints run a 60 per 60 s per-IP bucket and the named AI crawlers are exempt. An agent author needs that, so it is stated. - One of the 25 retired benches redirects to a successor instead of answering 410: polymarket-resolution-delay 308s to pm-resolution-delay. The block row, the routing bullet and the convention all say both now. - "Every citable surface carries asOf and dateModified" is true of the JSON and false of the badge SVG and the share-card PNG. Narrowed to the JSON, and freshness is documented as epoch milliseconds. - rankings[].ms holds the raw series value in the benchmark's stored unit, not milliseconds: bridge-fee puts a percent there. Commented, because that is a field someone writes code against. - The identical-conditions rule has documented exceptions (token-scoped feeds filtered to a pool, dRPC kept in rpc-cost by decision), so it now says split apart or filtered to a common scope. - The hand-typed category list named On-ramps and Explorers, which have no published benchmark, in an order that was wrong for what is published, and sat outside the generated markers where it would drift again. Deleted. And the one that matters most. The draft said "no provider pays for placement" and "nothing here is for sale" next to a repository that carries ten referral links in src/data/provider-registry.ts, rendered rel="noopener sponsored", and a funding.json declaring an Optimism Retro Funding project. That is the sentence a hostile reader screenshots. It is replaced by a Funding and links section that points at both artifacts, and by a narrower claim: placement is not for sale and a provider cannot change a figure, which is true by construction because a referral link is a display field and every number comes from PromQL in a spec. Who receives a commission is deliberately not stated here; that is Florent's to decide, not mine. Also fixed: the preview claim, since staging is a public SSO-free URL and providers are shown facts before publication by design; video/ described as the renderer behind the Export video button rather than internal tooling; the FAQ pointing at the provider-correction issue template; the keyed cohort measures paid plans, so "public endpoints" became endpoints any customer can reach; and an undocumented claim that corrections are logged, dropped because no log exists. Checks: 35 absolute URLs and 8 local paths fetched again, zero failures. No em or en dashes. The mermaid block renders: validated by a server-side render returning an 18 KB SVG with every label present. * round 4: the funding wording Florent settled, and data/ was never in the repo Florent's ruling on the money claim: nobody funds the project, the outbound links are affiliate links, and neutrality holds anyway. So the section says that plainly instead of the "nothing here is for sale" it replaced, and it carries no typed count of links, because openchainbench-e9 found that count had already moved today when Gains was added. It also now discloses what the specs already disclose and the section would have looked selective for omitting: nine keyed RPC benchmarks state that Chainstack, GetBlock and QuickNode provided the endpoints they are measured on, with the plan named. Verified at benchmarks/ethereum-rpc.yml line 29 and in eight sibling specs. Disclosing it next to the affiliate links is the difference between a disclosure and a selection. I was wrong about data/ and e9 was right. git ls-tree does not list it on dev or on this branch, and git check-ignore points at .gitignore line 44. It exists in my working tree only because prebuild wrote it. I had read the filesystem and believed I was reading the tree, which is the same mistake as trusting a 200 from a link checker: the command answered, but not the question I asked. The layout no longer mentions it, because a cloner never sees it. Four more claims narrowed to what is actually true: - BibTeX and RIS exports carry a URL and a retrieval date. They carry no asOf and no licence, so the citable-surface bullet now separates the JSON surfaces from the exported citations. - The markdown rewrite rule tests that Accept contains text/markdown and does NOT contain text/html. It is not about which comes first, so the sentence says what the rule actually tests. - The concept DOI resolves to the latest version of the record, which is v1.0.1 from 2026-06-22, four months old. Saying only "always the latest" invites someone to cite it for a figure they read today, so the line now gives the version and sends a current figure to the page and its asOf. - "Every published figure comes from the PromQL in a benchmark's spec" was false for the site as a whole: chain TVL, the perp venue context and the fee comparison are fetched from third-party APIs with no Prometheus in the path. Narrowed to every benchmark figure, which is the claim the affiliate argument actually needs. The stats block is now the one #3292's renderBlock emits, with the two category rows. The per-category list sums to 246, the same number the reconciliation sentence derives independently, and #3292 tests that sum. Checks: 35 absolute URLs and 7 local paths, zero failures. No em or en dashes. 273 lines. * round 5: the licence pointed at a file that does not contain it, and the funding paragraph contradicted itself Three factual defects that four rounds of checking had walked past, all found by openchainbench-e9 and verified here before changing anything. LICENSE on dev is a plain MIT text. It never mentions CC-BY-4.0 and never mentions data, so the line "Dataset CC-BY-4.0, attribution required. See LICENSE" sent a reader to a file that does not contain the licence it cites. Inside the repo that licence exists only as the `license` field on every API response, so that is what the README now points at. A LICENSE-DATA file would be the stronger artifact and is on Florent's list. The funding paragraph said "no sponsor and no grant pays for this project" and then, two sentences later, that the repository declares an Optimism Retro Funding project. Retro Funding is a grant programme, so the paragraph argued with itself. funding.json is a registration rather than money received, and it now says so. The absolute claim is held out of the file until Florent reconciles it: he told me nobody funds the project, and he told e9 on 2026-10-23 that two measured providers do. Both cannot be true, and it is the most quotable sentence in the file, so it waits for him rather than for me. What stays is what is verifiable either way: the affiliate links, the registration, the keyed cohort's endpoint provision, and the reason none of them can move a rank. "Nothing on the page is typed by hand" is false. A spec carries a hand-written abstract, methodology and FAQ. It is the numbers that are never typed, so that is what it claims now. Two sentences still carried the broad PromQL claim that round 4 narrowed in a third. The site publishes figures with no Prometheus in the path: chain TVL from api.llama.fi, the perp venue context, the fee comparison. All three now say benchmark figures. And "metrics a harness in this repository produced" became "metrics produced by the harnesses in this repository", because I cannot assert byte-identity between seventy deployed images and this source, and this session began by finding an image that was eighteen days behind it. Reader-facing, from e9's fresh-reader pass: - The first screen said what the project is and how to call it, but never what question it answers. One sentence now names two, both pointing at benchmarks that exist. - The headline paragraph was repeated two paragraphs later. Folded into one. - A live card sits under the badges, linking to the benchmark it shows, with a note that GitHub caches it. A stranger now sees what a benchmark looks like before reading a table. - "Every JSON figure carries an asOf ... in JSON, Markdown, RSS and an MCP server" was garbled: a JSON figure cannot be in Markdown. Formats and fields are separate sentences now. - "Not a preview service" is a phrase nobody has heard and read as defensive. It leads with the rule instead: not editable by the measured. - The routing bullet was five clauses joined by "and". It is three rules. - "usually a full measurement window and an audit round" overstated two conditions, neither of which is that pair. "For example" with three. - "Providers open PRs here" is a claim about history I could not check. "Providers can open PRs here" is true by policy. Checks: 36 absolute URLs and 7 local paths, zero failures, including the new card at /api/og/aggregator-head-lag (200, image/png, 59 KB). No em or en dashes. 277 lines. * narrow the last broad claim: benchmark figures, not every figure the site shows The citable-surface bullet said every figure is available as JSON, Markdown, RSS and MCP. True of benchmark figures; the site's non-benchmark numbers, the chain TVL and the perp venue context, are on none of those. Same narrowing already applied to the two PromQL sentences and the funding argument, so the file is now consistent about which figures it is making claims for. Last item from openchainbench-e9's final re-run, which otherwise found no false claim: 56 URLs all answering, 9 local paths present, the block test 8 pass, readme:sync --check current, no em or en dashes. * state the funding plainly: there is none Florent confirmed it on both channels, to me and to openchainbench-e9 when it put the reconciliation question to him: no funding. That settles the conflict round 5 held the sentence for, where his instruction to me said nobody funds the project and e9's note of 2026-10-23 said two measured providers did. The later statement supersedes it and the two channels now agree, so the absolute claim goes in. funding.json is described as a registration and not a payment received, which is entailed by "no grant funds this project" rather than inferred beside it. That was the internal contradiction in the paragraph: a sentence denying grants sitting above a declared grant programme. Everything else in the section stays as it was verified: the affiliate links read in one place only, the keyed cohort's endpoint provision disclosed in nine specs, and the argument that none of it reaches a measurement. * drop the funding section and every sentence touching it Florent's call: say nothing about funding in the README, no related sentence. So the Funding and links section is gone, and with it the affiliate-link disclosure, the funding.json mention and the keyed endpoint-provision note. Two sentences elsewhere pointed at it and are rewritten rather than left dangling. The scope bullet no longer says we do not sell placement and no longer refers the reader to a section below; it is now just "not a token, a fund, or an RPC vendor", because the claim that nobody outside the method can change a number already stands on its own in the next bullet and was saying the same thing twice. Nothing measured changes, and nothing is left half-said: a grep for sponsor, affiliate, commission, grant, investor, referralUrl, funding.json and "for sale" returns nothing. The keyed cohort's endpoint provision stays disclosed where it already was, in the methodology of each of the nine keyed specs. Checks: 34 absolute URLs and 7 local paths, zero failures. No em or en dashes. 269 lines.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #2342 (dev squash 2253521) onto main, nothing else from dev. One follow-up commit regenerates the literal middleware matcher from main's
removed-benches.ts(dev hasperp-pf-ratioheld back, main does not; the lockstep test caught it).Conflict resolved:
src/app/speedtest-rpc/opengraph-image.tsxis dev-only, kept deleted on main.Verified on this branch: typecheck 0 errors, bun 247/247, validate 220 specs, production
next buildexit 0, no static-to-dynamic errors. Every changed file already exists on main except the 5 new lib/test files.Why
Vercel bill. Root causes found from prod runtime logs + the codebase (no billing dashboard access, so this targets the mechanisms, not $ lines):
/api/internal/revalidate-aggregateevery 60 s →revalidateTag("benchmarks")→ every ISR page invalidated each minute. Every crawler hit = full render (x-vercel-cache: STALEon every page). Therevalidate = 3600on pages was meaningless.SET NX EX) to one purge perREVALIDATE_MIN_INTERVAL_SEC(default 600). Bench pages get a realrevalidate = 600. Data caches already refresh on their own 60-300 s timers.revalidate→max-age=0, satori (1-2 s CPU) on every share/crawler fetch (/products/x/opengraph-image= MISS, 2 s, on prod).revalidate = 86400on all 41; explicit headers aligned./benchmarks,/products,/answers,/comparerequest (~50 % of traffic, billed per edge invocation).removed-benches.tsby a test). Renamed bench →next.configredirects(); query-strip → inside the 3 dynamic API handlers; compare canonicalisation was already in the page. Verified in a prod build that lowercase live pages never invoke it.maxDuration = 300on pages (legacy from Prom-direct era);/api/aggregate70 s × 2 attempts. Provisioned memory is billed on wall time.maxDuration 90.buildProvidersresult (2.2 MB) never stored byunstable_cacheon Vercel → rebuilt on every render; bench page loaded the 8 MB aggregate just for a related-benches rail.providers.json(normalized wire format, ~1 MB so the data cache holds it) andindex.json(~100 KB). Site prefers them, falls back to the old path until the worker is redeployed. Honest note: the index build itself is 9 ms; the win is the aggregate fetch/parse.Not in this PR (needs owner / other repos)
@vercel/analytics+ Speed Insights (PostHog already loaded).telos+graphql-terminal: 3 316 preview builds / 30 d, no Ignored Build Step.Verified
bun test259 / typecheck / validate / lint (only the 4 pre-existingdevfiles).next start: all pages static (x-nextjs-cache: HIT), 0 static→dynamic errors, OG routes ISR-cached,/products/Alchemy→ 308,/benchmarks/network-coverage→ 308, hook: 1st call revalidates, 2nd returns{throttled:true}.cache: "no-store"on the providers fetch flipped pages to dynamic (hence the normalized <2 MB format);Number("")= 0 disabled the throttle when the env var was unset.After merge
The throttle, OG caching, middleware and maxDuration changes are effective as soon as this deploys.
providers.json/index.jsonstart being used when the VPS worker is redeployed fromdev; nothing breaks before that.🤖 Generated with Claude Code
https://claude.ai/code/session_01L4rjxemTvgRwrCG6D2iK5D