Repository navigation
bench 309 audit round: badges name the slice, prices read "latest value" - #3361
Merged
Merged
Conversation
The first audit found every bill right to the cent (163 of 163 live series equal the curve) and three presentation faults. All three fixed: - A "#1" earned on a bench's default slice was printed as "#1 overall". On wallet-cost Reown's badge said "cheapest embedded wallet" while it leads 2 of 9 cells; rpc-cost has the same shape. New defaultViewLabel names the slice ("Consumer app, 5 transactions a user · 10,000 users") on the badge SVG and the product page's badge cards, for any bench ranked per kind, venue or bucket. Appearances carry it as `defaultView` (providers-v28). - A monthly price was labelled "p50 · 24h" on the product page and badge. Benches whose value is a latest reading now say "latest value". - The calculator reads the committed curve and the ledger reads the harness, whose catalogue is baked into its image. The harness now publishes wallet_cost_catalogue_info{sha256,as_of}; the README says to compare it with the curve's catalogueSha256 after a price change. And the copy: `pnpm validate` now runs the independent scorer (proved to fire on a $1 shift), so the methodology's claim is true; Openfort's alternative reading is spelled out with its effect ($381, fourth rather than third); "six of the thirteen pages"; "sets a drift gauge" rather than "raises an alarm"; Turnkey's wallet cap counts all wallets held; no hard-coded date in the FAQ; "list price" in the SEO title; series and sample_size collapse the plan label. Harness: artifact gauges swap after the fetches instead of sitting empty during them; a gauge volume outside the curve range fails validation. Also from the staging check: the unit no longer truncates on mobile rows, and the "Not priced" reasons keep their capitals. Cache keys bench-unfiltered-v111, all-benchmarks-v104, bench-filters-v50.
Flotapponnier
added a commit
that referenced
this pull request
Oct 29, 2026
, #3369) (#3374) * bench 309 wallet-cost: what thirteen embedded wallet vendors bill the same app (#3357) * bench 309 wallet-cost: what thirteen embedded wallet vendors bill the same app A new bench and the first in the Wallets category. Privy, Dynamic, Turnkey, Web3Auth, Magic, Para, Coinbase CDP, thirdweb, Openfort, Circle, Reown, ZeroDev and Crossmint, priced for three usage profiles (5, 25 and 100 transactions a user) at any size from 100 to 1,000,000 monthly active users. The finding is the billing unit. Half the cohort bills per active user and nothing per transaction, half per signature, operation or credit, so the order changes with how often users sign: at 100,000 users Dynamic is $724 on every profile while Coinbase CDP goes from $5,075 to $100,000. harnesses/wallet-cost: a human-verified catalogue of every plan, with the page and date it was read, and a Go model that applies each vendor's meters, graduated tiers, caps, quote thresholds, minimums and own exclusions (Reown sends exchanges to a custom plan). Validation fails closed on the shapes that publish a plausible wrong price: a missing rate, a price on a quote, an unbounded meter, a misspelt key. Where a page reads two ways the plan is badged, scoped to the volumes the ambiguity affects. The calculator reads a committed curve with a point at every tier boundary, cap, minimum kink and plan crossover, and each point names the plan for the segment after it, so the slider is exact. verify/scorer.cjs is a second implementation in another language: it agrees with the harness on all 337 points and 1,170,000 random app sizes, to the cent and on the plan, and a mutation test shows it catches a $1 shift, a missing corner and a wrong plan. `pnpm validate` refuses a curve built from a different catalogue byte for byte. Freshness hashes each page's pricing lines, not its body: six of the thirteen pages change on every request while their prices do not. Gated on DEV_ONLY_BENCH_SLUGS until the harness has been live a cycle and an audit round has run. Eleven new marks and registry entries; Para is `getpara` because `para` is already a Hyperliquid deployer on bench 035. Cache keys bench-unfiltered-v109, all-benchmarks-v102, bench-filters-v48, providers-v27. * wallet-cost: the scorer is an ES module, as CI's lint requires (cherry picked from commit 8418f3a) * bench 309 audit round: badges name the slice, prices read "latest value" (#3361) The first audit found every bill right to the cent (163 of 163 live series equal the curve) and three presentation faults. All three fixed: - A "#1" earned on a bench's default slice was printed as "#1 overall". On wallet-cost Reown's badge said "cheapest embedded wallet" while it leads 2 of 9 cells; rpc-cost has the same shape. New defaultViewLabel names the slice ("Consumer app, 5 transactions a user · 10,000 users") on the badge SVG and the product page's badge cards, for any bench ranked per kind, venue or bucket. Appearances carry it as `defaultView` (providers-v28). - A monthly price was labelled "p50 · 24h" on the product page and badge. Benches whose value is a latest reading now say "latest value". - The calculator reads the committed curve and the ledger reads the harness, whose catalogue is baked into its image. The harness now publishes wallet_cost_catalogue_info{sha256,as_of}; the README says to compare it with the curve's catalogueSha256 after a price change. And the copy: `pnpm validate` now runs the independent scorer (proved to fire on a $1 shift), so the methodology's claim is true; Openfort's alternative reading is spelled out with its effect ($381, fourth rather than third); "six of the thirteen pages"; "sets a drift gauge" rather than "raises an alarm"; Turnkey's wallet cap counts all wallets held; no hard-coded date in the FAQ; "list price" in the SEO title; series and sample_size collapse the plan label. Harness: artifact gauges swap after the fetches instead of sitting empty during them; a gauge volume outside the curve range fails validation. Also from the staging check: the unit no longer truncates on mobile rows, and the "Not priced" reasons keep their capitals. Cache keys bench-unfiltered-v111, all-benchmarks-v104, bench-filters-v50. (cherry picked from commit fa4d1fd) * bench 309 round two: unpriced vendors keep their reason on every tab, and a price change alerts (#3365) The trading tab of the variant API returned nine rows with nothing said about Reown, Turnkey, ZeroDev and Crossmint, because the loader backfills a missing provider only on the unfiltered view. A bench can now declare that its not-served label holds on every tab (notServedOnEveryView); wallet-cost does, with the label reworded to name no size, and its filtered views list the missing vendors unranked with it. rpc-cost is left out: its label names the Ethereum dapp mix, which is false on its Solana tab. harnesses/wallet-cost/alerts.yml: WalletPricingArtifactDrift and WalletPricingArtifactUnreachable, deployed to the VPS rules directory. promtool showed the min_over_time(...[6h]) == 1 form used by rpc-cost's drift rule fires after 30 minutes on a series younger than 6h, so both use for: 6h, tested with promtool on a 7h drift (fires), a 2h drift that clears (silent) and a page that stops answering (fires). The rpc-cost rule on the VPS was corrected the same way. The copy now says a held change alerts. Cache keys bench-unfiltered-v112, all-benchmarks-v105, bench-filters-v51. (cherry picked from commit 62418b1) * bench 309 round three: every catalogue figure is under the drift check, and each unpriced vendor says why (#3367) Second audit: all 117 vendor-by-tab cells right to the cent, two majors. Privy's pay-as-you-go schedule ($2,000 base, $0.05 per user, $0.01 per signature) sits in one FAQ paragraph over the 300-character cap, so it was never hashed and six published bills could change with no drift. The reducer now splits text at sentence and clause ends before the cap, keeps bare band ranges ("1,001-5,000") and "up to" lines, and lets a source pin a phrase whose sentence carries a rule but no number (Reown's "custom plan"). WALLET_COST_PIN=1 now lists every catalogue figure missing from a page's hashed lines; it is empty for all thirteen. Hashes re-pinned after confirming zero drift on all thirteen over the previous six hours; stable over three fetches. The trading tab said Reown had "no self-serve price for this app size", while it prices the same size on the other profiles; Crossmint read the same on 1k tabs where its real limit is transactions. The loader now takes each vendor's reason for the view from the curve (notServedReason), the same reason the calculator shows. Also: WalletCostSiteCurveMismatch, from a new wallet_cost_site_curve_matches{branch} gauge comparing the curve on the dev and main branches with the harness's own catalogue (promtool-tested, live); Privy's Scale price at exactly 10,000 users is badged (its card reads 2,500 to 9,999) and the copy states the other reading ($2,000, spread 9.3x); validate's scorer sample stated as 78,000 sizes; "badged in the calculator"; the validator refuses a polled source without a recorded hash; the freshness timestamp is set when prices are computed, not after the page fetches. Cache keys bench-unfiltered-v114, all-benchmarks-v107, bench-filters-v53. (cherry picked from commit c4c3da3) * bench 309 SEO round: pricing pages for the vendors, a sitemap hub gate, prose checked against the model (#3369) SEO/GEO audit of bench 309 on staging, verdict fix then ship: - Production's sitemap listed /benchmarks/category/wallets, which 404s there: category hubs were built from the dev worker's blob without the "does this build serve the bench" gate the bench routes have had since arc-rpc. Same gate now, with a source test that fails without it. - Eleven vendors measured only by bench 309 had thin product pages with one unsized figure ("ranks #9 of 10 at $499"). Their title is now "<Vendor> pricing 2026: monthly bill from 100 to 1M users", the description carries the 10,000-user consumer bill and rank, and every wallet-cost vendor's page (thirdweb and Coinbase CDP included) gets a server-rendered table of its bill and plan for each profile and size, read from the same curve as the calculator, ranked on the cent so ties share a rank. - A product measured in several categories leads its title and snippet with the category it is measured most in: thirdweb's single wallet rank no longer displaces thirty-four RPC benches, and the title and description no longer disagree (unranked rows also no longer pick the category). - The prose types bills and winners by hand. `pnpm validate` now checks every dollar figure in the findings, FAQ, methodology, intro and description against the bills the model produces at the published volumes and the catalogue's fees and rates; proved to fire on $105,000 for $106,000. Volume thresholds ("$1M") are skipped. - Findings render on the page under their own H2, after the grid. - seo_title "Embedded wallet pricing 2026: cheapest vendor per app size"; title without a typed count; metric "Monthly cost for a consumer app at 10,000 users" (the comma broke every citable sentence); description and intro open on {{best_name}} at {{best_p50}}; abstract uses {{declared_count}}; the Openfort reading is qualitative in the methodology and quantified in the findings. Cache keys bench-unfiltered-v115, all-benchmarks-v108, bench-filters-v54. (cherry picked from commit 8fb9658) * release bench 309 wallet-cost: un-gate, cache keys numbered for main Leaves DEV_ONLY_BENCH_SLUGS after a day of harness cycles and three audit rounds whose findings are all fixed (#3361, #3365, #3367, #3369). Cache keys on this branch's own sequence: bench-unfiltered-v106, all-benchmarks-v99, bench-filters-v46, providers-v22; provider-set hash recorded. README stats synced.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First audit of bench 309 (checks + two passes): every bill right to the cent, 163/163 live series equal to the curve, scorer 0 disagreements. Three presentation majors, all fixed here:
defaultViewLabelnow names the slice on the badge SVG and on product-page badge cards for any bench ranked per kind / venue / bucket, rpc-cost included. Appearances carrydefaultView(providers-v28).wallet_cost_catalogue_info{sha256,as_of}; README step to compare it with the curve after a price change.Copy and minors: validate now runs
verify/scorer.mjs(proved to fire); Openfort's alternative reading and its rank effect stated; "six of the thirteen pages"; drift gauge wording; Turnkey wallet cap note; no hard-coded FAQ date; "list price" SEO title;min by (provider)on series/sample_size; artifact gauges swap after fetching; volume range check. Mobile: unit no longer truncated; reasons keep their capitals.bun test 753 pass, validate, tsc, eslint 0 errors, go test.
After merge: harness rebuild (catalogue_info gauge), worker rebuild (spec copy +
defaultView).