Repository navigation
bench №004 metadata coverage - #8
Merged
Merged
Conversation
…feeder + codex jwt fix)
bakabhai993
pushed a commit
to bakabhai993/OpenChainBench
that referenced
this pull request
Sep 12, 2026
bench №004 metadata coverage
Flotapponnier
added a commit
that referenced
this pull request
Sep 22, 2026
…from 404ing the best converting page, retire the second dead answer (#2625) Third audit. Two of these are mine. #2620 moved the registry one-liner in front of the measured lead in the product description. Registry descriptions run to a median of 166 characters against a 155 cap, so capSnippet dropped the lead entirely and every one of the 220 product snippets lost its numbers: /products/invo went from "Invo ranks #7 of 104 on trading builder fees collected at $12.8K, #8 of 11 on Crypto trading app iOS App Store ratings at 4.23x (p50, 24h)." to a vendor blurb with nothing measured in it. The vendor's own SERP result already says what the product is; the ranks are the only thing ours carries that theirs does not. A one-liner under 80 characters still leads; a longer one follows the ranks, trimmed to its first clause. #2618 turned the compare thin gate from noindex into notFound. That is right for a pair that structurally has nothing to compare and wrong for one whose data dipped for a render window: /compare/fomo-vs-invo sits at exactly two live shared benches and earns 36 of the site's 289 clicks, so one flapping bench would have 404ed the best converting URL on the property. The 404 now keys on the number of SHARED benches, which is stable, and a live-count dip goes back to noindex, which recovers. The sitemap's 3/4 buffer in adhoc-pairs.ts stays exactly as written: it exists for this flap and its comment says so. The audit proposed lowering it to match the render floor; that would have removed the buffer precisely when crossing it became expensive. And the same retired-bench class as yesterday, second instance: /answers/polymarket-vs-kalshi-fees references pm-fee-comparison too, 404s too, and was advertised by the same hub. The filter added yesterday works; the slug was missing from the set. Its YAML also hand-types "flat 2%", "roughly $0.07" and a "$0.20 / $0.80" crossover, so it must not be revived as written. Claude-Session: https://claude.ai/code/session_01L4rjxemTvgRwrCG6D2iK5D Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier
added a commit
that referenced
this pull request
Oct 30, 2026
…lculators off every bench page Fixes from the 2026-10-30 re-audits of benches 309 and 310 (data and SEO passes), applied to bench 312 as well since it shares the code. Ranks. Product pages ranked only the prices that hold while the ledger and the calculator rank every priced row except an "at least" price at or under the leader, so /products/portal read "#8 of 8" in its bench list and "#6 of 6" in the table under it, and Privy "#7 of 8" beside "not ranked". src/lib/cost-rank.ts now holds the one rule; the product tables and the product metadata use it, with the same denominator. Machine surfaces. /api/stat ignored ?bucket=, so ?kind=trading&bucket=100k answered with the 10k cell while echoing only the kind; it now applies the size. /api/stat and the MCP rankings carry priceReading ("at most", "at least"), so Privy's floor is no longer a plain $6,500 in JSON. The variant API copies the headline seoTitle and seoDescription instead of resolving them on a filtered view under the headline's label. A new spec field, headline_slice, replaces "measured on AI agents, 50 transactions each, Medium size" in the citable headline with what the cell is. The Dataset JSON-LD of a bench with ledger_reliability: false no longer lists sample_size or percentile axes. Weight. Every bench page downloaded the cost calculators and their curves (68 KB gzip) because the shared route imported them statically; they now load through next/dynamic from a client wrapper, SSR kept. Display. The calculators print cents when a bill has cents (Dfns $2,916.67, not $2,917); product appearance rows print a published price whole ($51,299, not $51.3K); a $0 floor reads "$0 on its published rates" instead of "at least $0". Vendors priced only by the pricing benches get an H1 that says pricing, and one on several of them (Circle, Turnkey, Openfort, ZeroDev) a title naming every kind and a description from the first headline bill it has, instead of bench 309's alone. Copy. 309: Portal's meter in the Units bullet and the transactions FAQ, the changelog sentences out of the Dynamic finding and the cohort, the false Openfort cross-reference dropped, Privy's 100k bills read from the curve, the freshness gap stated. 310: the bot FAQ and finding name the curve's winners (Magic ties Circle at 10,000 transactions), Dfns and Crossmint in Units, Circle's chain scope and USDC rebate, bot and payout cells in the intro, a title that does not promise one winner. Validator. A question asking for the cheapest must be answered by a rank cell in its first sentence, and only a rank cell exempts a winner sentence (any vendor cell did). Proved by putting the old bot answer back: it fails. Harnesses (309, 310, 312). The site curves are re-read every 5 minutes and the mismatch alert fires after 15 (was hourly and 2h, blind to the 30 minutes production priced from two catalogues on 2026-10-30); the release order is in the wallet-cost README. Cache keys bench-unfiltered-v131, all-benchmarks-v124, bench-filters-v69.
Flotapponnier
added a commit
that referenced
this pull request
Oct 30, 2026
…lculators off every bench page Fixes from the 2026-10-30 re-audits of benches 309 and 310 (data and SEO passes), applied to bench 312 as well since it shares the code. Ranks. Product pages ranked only the prices that hold while the ledger and the calculator rank every priced row except an "at least" price at or under the leader, so /products/portal read "#8 of 8" in its bench list and "#6 of 6" in the table under it, and Privy "#7 of 8" beside "not ranked". src/lib/cost-rank.ts now holds the one rule; the product tables and the product metadata use it, with the same denominator. Machine surfaces. /api/stat ignored ?bucket=, so ?kind=trading&bucket=100k answered with the 10k cell while echoing only the kind; it now applies the size. /api/stat and the MCP rankings carry priceReading ("at most", "at least"), so Privy's floor is no longer a plain $6,500 in JSON. The variant API copies the headline seoTitle and seoDescription instead of resolving them on a filtered view under the headline's label. A new spec field, headline_slice, replaces "measured on AI agents, 50 transactions each, Medium size" in the citable headline with what the cell is. The Dataset JSON-LD of a bench with ledger_reliability: false no longer lists sample_size or percentile axes. Weight. Every bench page downloaded the cost calculators and their curves (68 KB gzip) because the shared route imported them statically; they now load through next/dynamic from a client wrapper, SSR kept. Display. The calculators print cents when a bill has cents (Dfns $2,916.67, not $2,917); product appearance rows print a published price whole ($51,299, not $51.3K); a $0 floor reads "$0 on its published rates" instead of "at least $0". Vendors priced only by the pricing benches get an H1 that says pricing, and one on several of them (Circle, Turnkey, Openfort, ZeroDev) a title naming every kind and a description from the first headline bill it has, instead of bench 309's alone. Copy. 309: Portal's meter in the Units bullet and the transactions FAQ, the changelog sentences out of the Dynamic finding and the cohort, the false Openfort cross-reference dropped, Privy's 100k bills read from the curve, the freshness gap stated. 310: the bot FAQ and finding name the curve's winners (Magic ties Circle at 10,000 transactions), Dfns and Crossmint in Units, Circle's chain scope and USDC rebate, bot and payout cells in the intro, a title that does not promise one winner. Validator. A question asking for the cheapest must be answered by a rank cell in its first sentence, and only a rank cell exempts a winner sentence (any vendor cell did). Proved by putting the old bot answer back: it fails. Harnesses (309, 310, 312). The site curves are re-read every 5 minutes and the mismatch alert fires after 15 (was hourly and 2h, blind to the 30 minutes production priced from two catalogues on 2026-10-30); the release order is in the wallet-cost README. Cache keys bench-unfiltered-v132, all-benchmarks-v125, bench-filters-v70.
Flotapponnier
added a commit
that referenced
this pull request
Oct 30, 2026
…lculators off every bench page (#3433) Fixes from the 2026-10-30 re-audits of benches 309 and 310 (data and SEO passes), applied to bench 312 as well since it shares the code. Ranks. Product pages ranked only the prices that hold while the ledger and the calculator rank every priced row except an "at least" price at or under the leader, so /products/portal read "#8 of 8" in its bench list and "#6 of 6" in the table under it, and Privy "#7 of 8" beside "not ranked". src/lib/cost-rank.ts now holds the one rule; the product tables and the product metadata use it, with the same denominator. Machine surfaces. /api/stat ignored ?bucket=, so ?kind=trading&bucket=100k answered with the 10k cell while echoing only the kind; it now applies the size. /api/stat and the MCP rankings carry priceReading ("at most", "at least"), so Privy's floor is no longer a plain $6,500 in JSON. The variant API copies the headline seoTitle and seoDescription instead of resolving them on a filtered view under the headline's label. A new spec field, headline_slice, replaces "measured on AI agents, 50 transactions each, Medium size" in the citable headline with what the cell is. The Dataset JSON-LD of a bench with ledger_reliability: false no longer lists sample_size or percentile axes. Weight. Every bench page downloaded the cost calculators and their curves (68 KB gzip) because the shared route imported them statically; they now load through next/dynamic from a client wrapper, SSR kept. Display. The calculators print cents when a bill has cents (Dfns $2,916.67, not $2,917); product appearance rows print a published price whole ($51,299, not $51.3K); a $0 floor reads "$0 on its published rates" instead of "at least $0". Vendors priced only by the pricing benches get an H1 that says pricing, and one on several of them (Circle, Turnkey, Openfort, ZeroDev) a title naming every kind and a description from the first headline bill it has, instead of bench 309's alone. Copy. 309: Portal's meter in the Units bullet and the transactions FAQ, the changelog sentences out of the Dynamic finding and the cohort, the false Openfort cross-reference dropped, Privy's 100k bills read from the curve, the freshness gap stated. 310: the bot FAQ and finding name the curve's winners (Magic ties Circle at 10,000 transactions), Dfns and Crossmint in Units, Circle's chain scope and USDC rebate, bot and payout cells in the intro, a title that does not promise one winner. Validator. A question asking for the cheapest must be answered by a rank cell in its first sentence, and only a rank cell exempts a winner sentence (any vendor cell did). Proved by putting the old bot answer back: it fails. Harnesses (309, 310, 312). The site curves are re-read every 5 minutes and the mismatch alert fires after 15 (was hourly and 2h, blind to the 30 minutes production priced from two catalogues on 2026-10-30); the release order is in the wallet-cost README. Cache keys bench-unfiltered-v132, all-benchmarks-v125, bench-filters-v70.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the fourth published benchmark — token metadata coverage — and syncs the underlying aggregator harness with two upstream fixes from mobula-api.
Bench № 004 — Aggregator Token Metadata Coverage
For freshly-launched tokens (sourced from Mobula Pulse V2 across pump.fun, Meteora DBC, Four.meme, Raydium CPMM, Zora, BaseApp, Bags, Moonshot), measures what share of four canonical fields (
logo,description,twitter,website) each aggregator returns.benchmarks/metadata-coverage.yml/api/2/token/details), Codex (GraphQLtokenquery), Jupiter (/v6/tokens, Solana only)metadata_coverage_success_total / metadata_coverage_checks_total, expressed as percentquantile_over_timeon rolling 1h rates so p50/p90/p99 reflect drift over the dayHarness sync
Picks up two fixes from
mobula-api/miniapps/aggregator-latency-benchmark:mobula_pulse_monitor.go(new file) — slim Pulse V2 feeder. Subscribes to Pulse for new launchpad tokens and pushes them onto the metadata-coverage queue. Emits no Pulse-specific metrics (Pulse is not a benched aggregator). Without this, the metadata-coverage gauges have no fresh tokens to check.metadata_coverage_monitor.go— Codex GraphQL no longer accepts the raw Defined session cookie as Bearer (returns 401). Now mints a JWT from the cookie via defined.fi/api before calling Codex — same path used byhead_lag_monitor.go.main.go— wires the new Pulse feeder goroutine.Sanity
pnpm validatepasses (4 specs valid)pnpm buildgenerates /benchmarks/metadata-coverage successfullyos.Getenv)What happens after merge