Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions benchmarks/aggregator-head-lag.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# OpenChainBench — Bench № 001
# OpenChainBench. Bench № 001

slug: aggregator-head-lag
number: "001"
Expand All @@ -23,7 +23,7 @@ methodology:
- "Regions: us-east, eu-west, sgp. Cross-region median reported in the headline."
- "Reference: archive nodes per chain, validated against block hashes."
- "Metric: gauge `head_lag_seconds`, sampled every 15 seconds. Aggregated over the 24-hour window using `quantile_over_time`."
- "Success rate: presence ratio — share of expected sampling slots where a value was actually emitted (5,760 expected per provider per day at the 15s cadence). 100% means the aggregator's feed was reachable for the full window."
- "Success rate: presence ratio. share of expected sampling slots where a value was actually emitted (5,760 expected per provider per day at the 15s cadence). 100% means the aggregator's feed was reachable for the full window."
- "Cardinality: 3 aggregators × 3 chains × 3 regions = 27 active series."

findings: []
Expand All @@ -35,8 +35,8 @@ prometheus:
window: 24h

# Real metric exposed by the head-lag harness:
# head_lag_seconds{aggregator, chain, region} — gauge in seconds
# head_lag_errors_total{aggregator, chain, region, error_type} — counter
# head_lag_seconds{aggregator, chain, region} . gauge in seconds
# head_lag_errors_total{aggregator, chain, region, error_type} . counter
#
# Label is `aggregator=`, not `provider=`. Values are in seconds; queries
# multiply by 1000 so the loader stores milliseconds (consistent across the
Expand Down
12 changes: 6 additions & 6 deletions benchmarks/bridge-fee.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# OpenChainBench — Bench № 003
# OpenChainBench. Bench № 003

slug: bridge-fee
number: "003"
Expand All @@ -10,11 +10,11 @@ metric: Effective fee
unit: pct

abstract: |
We measure the effective fee each bridge charges — the bottom-line cost a
user pays, fees plus slippage plus destination gas — as a percentage of
We measure the effective fee each bridge charges. the bottom-line cost a
user pays, fees plus slippage plus destination gas. as a percentage of
trade size. The harness uses Prometheus gauge `bridge_cost_percent`
rather than the narrower `bridge_fees_percent` because providers
structure pricing differently — some charge an explicit fee, others bake
structure pricing differently. some charge an explicit fee, others bake
it into the spread. Comparing only the explicit fee field would let
aggregators with implicit pricing show a misleading 0%. Sampled at $300
notional so small-amount fixed-fee blowups don't skew the percentile
Expand All @@ -25,7 +25,7 @@ methodology:
- "Reference notional: $300 USDC. Smaller sizes ($5, $50) are captured by the harness but excluded from this report."
- "Cost: `bridge_cost_percent{amount_usd=\"300\"}`, in percent of notional. Includes fees + slippage + destination gas."
- "Cadence: full sweep every 5 minutes for 24 hours."
- "Region: eu-west (single point — multi-region requires running additional monitor instances)."
- "Region: eu-west (single point. multi-region requires running additional monitor instances)."
- "Failures (quote_failed, unsupported route) excluded from cost aggregates and counted toward success rate."

findings: []
Expand All @@ -43,7 +43,7 @@ prometheus:
# Why cost rather than fee:
# bridge_fees_percent only captures the explicit fee field in each quote.
# Some providers (e.g. Mobula) bake the cost into the spread instead of
# reporting an explicit fee, so they would render as 0% — misleading
# reporting an explicit fee, so they would render as 0%. misleading
# readers. bridge_cost_percent = fees + slippage + gas, which is the
# number that actually leaves the user's wallet.

Expand Down
8 changes: 4 additions & 4 deletions benchmarks/bridge-quote-latency.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# OpenChainBench — Bench № 002
# OpenChainBench. Bench № 002

slug: bridge-quote-latency
number: "002"
Expand All @@ -21,7 +21,7 @@ methodology:
- "Routes: USDC pairs spanning Solana, Base and Arbitrum."
- "Notional sizes: $5, $50, $300 per quote."
- "Cadence: full sweep (4 routes × 3 amounts × N bridges) every 5 minutes for 24 hours."
- "Region: eu-west (single point — multi-region requires running additional monitor instances)."
- "Region: eu-west (single point. multi-region requires running additional monitor instances)."
- "Histogram buckets: 50, 100, 200, 500, 1000, 2000, 5000, 10000 ms."
- "Failures (quote_failed, execution_failed, unsupported route) excluded from latency aggregates and counted toward success rate."

Expand All @@ -35,8 +35,8 @@ prometheus:

# Real metrics exposed by the bridge-monitor harness:
# bridge_quote_latency_ms_{bucket,sum,count}{bridge, from_chain, to_chain, from_token, to_token, amount_usd, region}
# bridge_quote_success{...} — gauge 0/1
# bridge_errors_total{..., error_type} — counter
# bridge_quote_success{...} . gauge 0/1
# bridge_errors_total{..., error_type} . counter
#
# Label is `bridge=`, not `provider=`. Keep that in mind when extending.

Expand Down
16 changes: 8 additions & 8 deletions benchmarks/metadata-coverage.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# OpenChainBench — Bench № 004
# OpenChainBench. Bench № 004

slug: metadata-coverage
number: "004"
Expand All @@ -14,14 +14,14 @@ abstract: |
freshly-launched tokens. The harness subscribes to Mobula Pulse V2 for new
tokens on Solana, BNB Chain and Base, then asks each aggregator (Mobula,
Codex, Jupiter where the chain is supported) for that token's metadata and
records whether four canonical fields — `logo`, `description`, `twitter`
and `website` — were returned. The aggregate is the share of (provider,
records whether four canonical fields. `logo`, `description`, `twitter`
and `website`. were returned. The aggregate is the share of (provider,
chain, field) combinations that returned a populated value across all
fresh-token checks in the window.

Higher is better. A value of 100% means every field was returned for every
fresh token; a value of 50% means half the fields were missing on average.
Sort order on this page is ascending — the lowest coverage shows first,
Sort order on this page is ascending. the lowest coverage shows first,
which is the inverse of latency benchmarks. Readers should compare the
columns themselves; numerical p50 is the headline.

Expand All @@ -30,9 +30,9 @@ methodology:
- "Coverage check: for each fresh token, query each aggregator's metadata endpoint and record whether `logo`, `description`, `twitter` and `website` are populated."
- "Aggregators in scope: Mobula (`/api/2/token/details`), Codex (GraphQL `token` query, JWT-authenticated), Jupiter (`/v6/tokens/<mint>`, Solana only)."
- "Cadence: queue-driven; every newly-discovered token is checked once across the three aggregators. Steady-state ≈ several hundred checks per provider per hour."
- "Region: `eu-west` (single point — multi-region requires running additional monitor instances)."
- "Ratio: `metadata_coverage_success_total / metadata_coverage_checks_total`, expressed in percent. Failures of the metadata API itself (timeouts, 5xx) are not counted in the denominator — only successful responses with the field check applied."
- "Jupiter is Solana-only — it appears with zero coverage on EVM chains by construction; the cross-chain headline excludes Jupiter on chains it does not support."
- "Region: `eu-west` (single point. multi-region requires running additional monitor instances)."
- "Ratio: `metadata_coverage_success_total / metadata_coverage_checks_total`, expressed in percent. Failures of the metadata API itself (timeouts, 5xx) are not counted in the denominator. only successful responses with the field check applied."
- "Jupiter is Solana-only. it appears with zero coverage on EVM chains by construction; the cross-chain headline excludes Jupiter on chains it does not support."

findings: []

Expand All @@ -55,7 +55,7 @@ prometheus:
# / sum(rate(metadata_coverage_checks_total{provider="X"}[24h]))
#
# We use quantile_over_time on a rolling 1h rate to give p50/p90/p99 the
# same shape as latency benchmarks — so a coverage rate that drifts during
# same shape as latency benchmarks. so a coverage rate that drifts during
# the day shows tail behaviour, not a single flat value.

providers:
Expand Down
16 changes: 8 additions & 8 deletions src/app/about/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ export default function AboutPage() {
About OpenChainBench
</h1>
<p className="mt-4 text-xl sm:text-2xl text-ink-muted leading-snug">
Open, reproducible benchmarks for crypto infrastructure — measured in
Open, reproducible benchmarks for crypto infrastructure. measured in
the open, published in the same format every time.
</p>

Expand All @@ -23,25 +23,25 @@ export default function AboutPage() {
Crypto infrastructure runs the world&apos;s open financial rails, and almost none of it is benchmarked in public. Every aggregator, bridge and market-data feed quotes its own numbers, on its own terms, with its own regions and its own definitions of &ldquo;fast&rdquo;. OpenChainBench picks one definition at a time, runs the experiment in the open, and publishes the script alongside the result.
</p>
<p className="mt-4 text-base leading-relaxed text-ink-soft">
The goal is to make performance an observable property of crypto infrastructure — like uptime in SaaS, or p99 in databases — so builders can choose providers on data and users can hold them to it.
The goal is to make performance an observable property of crypto infrastructure. like uptime in SaaS, or p99 in databases. so builders can choose providers on data and users can hold them to it.
</p>

<SectionRule label="How" number="ii" />
<p className="text-base leading-relaxed text-ink-soft">
Every benchmark is a YAML spec plus a harness. The spec describes what to measure, which providers, which Prometheus queries hold the numbers; the harness runs continuously and exposes those metrics. A single shared Prometheus scrapes every harness; the site queries Prometheus directly and re-renders every minute. Every provider is rendered with equal visual weight — readers do their own ranking.
Every benchmark is a YAML spec plus a harness. The spec describes what to measure, which providers, which Prometheus queries hold the numbers; the harness runs continuously and exposes those metrics. A single shared Prometheus scrapes every harness; the site queries Prometheus directly and re-renders every minute. Every provider is rendered with equal visual weight. readers do their own ranking.
</p>
<p className="mt-4 text-base leading-relaxed text-ink-soft">
Anyone can submit a benchmark. The{" "}
<Link className="lnk" href="/contribute">contribution guide</Link>{" "}
walks through the steps. New providers, new metrics, new chains — all welcome via pull request.
walks through the steps. New providers, new metrics, new chains. all welcome via pull request.
</p>

<SectionRule label="What you can do" number="iii" />
<ul className="space-y-4 text-base leading-relaxed text-ink-soft">
<li className="flex gap-3"><span className="text-ink-muted">—</span><span><strong>Read</strong> the{" "}<Link className="lnk" href="/benchmarks">live benchmarks</Link>{" "}.</span></li>
<li className="flex gap-3"><span className="text-ink-muted">—</span><span><strong>Reproduce</strong> any number — the{" "}<Link className="lnk" href="/methodology">methodology</Link>{" "}page tells you how.</span></li>
<li className="flex gap-3"><span className="text-ink-muted">—</span><span><strong>Contribute</strong> a harness via{" "}<a className="lnk" href="https://github.com/OpenChainBench/OpenChainBench">GitHub</a>. Pull requests for new providers and new benchmarks are reviewed openly.</span></li>
<li className="flex gap-3"><span className="text-ink-muted">—</span><span><strong>Discuss</strong> ideas in{" "}<a className="lnk" href="https://github.com/OpenChainBench/OpenChainBench/discussions">GitHub Discussions</a>{" "}or follow{" "}<a className="lnk" href="https://twitter.com/openchainbench">@openchainbench</a>.</span></li>
<li className="flex gap-3"><span className="text-ink-faint">·</span><span><strong>Read</strong> the{" "}<Link className="lnk" href="/benchmarks">live benchmarks</Link>{" "}.</span></li>
<li className="flex gap-3"><span className="text-ink-faint">·</span><span><strong>Reproduce</strong> any number. the{" "}<Link className="lnk" href="/methodology">methodology</Link>{" "}page tells you how.</span></li>
<li className="flex gap-3"><span className="text-ink-faint">·</span><span><strong>Contribute</strong> a harness via{" "}<a className="lnk" href="https://github.com/OpenChainBench/OpenChainBench">GitHub</a>. Pull requests for new providers and new benchmarks are reviewed openly.</span></li>
<li className="flex gap-3"><span className="text-ink-faint">·</span><span><strong>Discuss</strong> ideas in{" "}<a className="lnk" href="https://github.com/OpenChainBench/OpenChainBench/discussions">GitHub Discussions</a>{" "}or follow{" "}<a className="lnk" href="https://twitter.com/openchainbench">@openchainbench</a>.</span></li>
</ul>

<SectionRule label="Contact" number="iv" />
Expand Down
10 changes: 5 additions & 5 deletions src/app/benchmarks/[slug]/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ export default async function BenchmarkPage({

{isDraft && <DraftNotice source={benchmark.source} />}

{/* Charts + tables — no Fig. captions, just data */}
{/* Charts + tables. no Fig. captions, just data */}
{!isDraft && (
<>
<div className="mt-12">
Expand Down Expand Up @@ -174,7 +174,7 @@ export default async function BenchmarkPage({
</div>
)}

{/* About — collapsed by default */}
{/* About. collapsed by default */}
<details className="mt-14 group border-t border-rule">
<summary className="flex cursor-pointer items-center justify-between py-4 list-none">
<span className="text-[11px] font-medium uppercase tracking-[0.18em] text-ink">
Expand All @@ -197,7 +197,7 @@ export default async function BenchmarkPage({
<ul className="space-y-2 text-sm leading-relaxed text-ink-soft">
{benchmark.methodology.map((m) => (
<li key={m} className="flex gap-3">
<span className="text-ink-faint mt-1.5">—</span>
<span className="text-ink-faint mt-1.5">·</span>
<span>{m}</span>
</li>
))}
Expand All @@ -215,7 +215,7 @@ export default async function BenchmarkPage({
/>
)}

{/* Source code link — bottom of page */}
{/* Source code link. bottom of page */}
{!isDraft && (
<p className="mt-4 text-[11px] uppercase tracking-[0.16em] text-ink-muted">
Source code{" "}
Expand Down Expand Up @@ -273,7 +273,7 @@ function DraftNotice({ source }: { source: string }) {
return (
<div className="mt-10 card p-6 text-center">
<p className="text-[10px] font-medium uppercase tracking-[0.16em] text-ink-faint">
Draft — no live data yet
Draft. no live data yet
</p>
<p className="mt-3 text-sm text-ink-muted">
The spec is published. Numbers will appear here as soon as the harness
Expand Down
2 changes: 1 addition & 1 deletion src/app/benchmarks/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ export default async function BenchmarksIndex() {

<div className="mt-20 card p-8 text-center">
<p className="text-base text-ink-muted">
{all.length} {all.length === 1 ? "benchmark" : "benchmarks"} live — and counting.
{all.length} {all.length === 1 ? "benchmark" : "benchmarks"} live. and counting.
</p>
<Link
href="/contribute"
Expand Down
Loading
Loading