From 0b14e88982b8746aa6e5b4ab57d4acee773a0ab9 Mon Sep 17 00:00:00 2001 From: Flotapponnier <160007691+Flotapponnier@users.noreply.github.com> Date: Fri, 9 Oct 2026 15:05:34 +0200 Subject: [PATCH 1/2] the bench said seven relays and one of them no longer exists (#2919) bloXroute's max-profit relay has no DNS record at all: NXDOMAIN at 8.8.8.8, so not a local resolver artefact. It delivered 0 payloads in the last 24h while the other six all moved (ultrasound 5841, titan 5523, aestus 2912, bloxroute-regulated 2813, flashbots 264, agnostic 131). The row looked alive for a precise reason worth recording: ebb_relay_payloads_total is a counter, and a counter never goes down, so it kept displaying 26,395 accumulated payloads long after the host stopped resolving. Reading the total says nothing; only increase() over a window does. The spec advertised "seven MEV-Boost relays" in four places, including the FAQ answer, and named bloXroute max-profit in two of them. Corrected to six. Safe to drop: no published panel or query pins a relay label, the counter is a methodology cross-check only, so no column goes null. go build and go vet clean, pnpm validate clean. --- benchmarks/evm-block-builders.yml | 10 +++++----- harnesses/evm-block-builders/cmd/script/relays.go | 9 ++++++++- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/benchmarks/evm-block-builders.yml b/benchmarks/evm-block-builders.yml index 191dd7777..aaa28fcce 100644 --- a/benchmarks/evm-block-builders.yml +++ b/benchmarks/evm-block-builders.yml @@ -30,8 +30,8 @@ seo_intro: | proposer built locally instead of outsourcing; that row is the live measure of how much of Ethereum still self-builds. As an independent cross-check the harness also polls the public bidtrace data APIs of - seven MEV-Boost relays (Flashbots, ultrasound, Agnostic, bloXroute - max-profit and regulated, Titan, Aestus) and counts delivered + six MEV-Boost relays (Flashbots, ultrasound, Agnostic, bloXroute + regulated, Titan, Aestus) and counts delivered payloads per relay, a signal that does not depend on any builder's self-chosen label. @@ -51,7 +51,7 @@ abstract: | bench 016. The headline metric is each builder's percentage of all blocks observed in the last 24 hours. Two companion measurements ship in the same harness. First, an independent cross-check: the public - bidtrace APIs of seven MEV-Boost relays are polled every five minutes + bidtrace APIs of six MEV-Boost relays are polled every five minutes and delivered payloads are counted per relay, deduped per relay by slot high-water-mark; the same slot appearing on several relays is normal because builders multi-home their bids. Second, an L2 @@ -68,7 +68,7 @@ methodology: - "Vanilla detection: empty extraData or execution-client default tags (geth, reth, nethermind, besu, erigon, go1.x, platform strings) count as builder=vanilla, meaning the proposer built the block locally instead of outsourcing to a builder. This row is the live self-built share of Ethereum." - "Unattributed handling: any non-empty tag not in the table counts as builder=other, increments ebb_unattributed_total, and logs the raw extraData string. The table grows from operator logs instead of silently under-counting, the bench 016 pattern." - "Self-labeling caveat: extraData is voluntary. A builder can rebrand, blank its tag, or impersonate another. The relay cross-check below is the independent signal; a large divergence between extraData share and relay-side delivered payloads would surface tag games." - - "Relay cross-check: seven MEV-Boost relay data APIs (Flashbots, ultrasound, Agnostic, bloXroute max-profit, bloXroute regulated, Titan, Aestus) are polled keyless every 5 min via GET /relay/v1/data/bidtraces/proposer_payload_delivered?limit=50. Delivered payloads are counted per relay (ebb_relay_payloads_total), deduped per relay by slot high-water-mark. The same slot on multiple relays is normal: builders multi-home bids, so relay counters measure relay share, not a partition of blocks." + - "Relay cross-check: six MEV-Boost relay data APIs (Flashbots, ultrasound, Agnostic, bloXroute regulated, Titan, Aestus) are polled keyless every 5 min via GET /relay/v1/data/bidtraces/proposer_payload_delivered?limit=50. Delivered payloads are counted per relay (ebb_relay_payloads_total), deduped per relay by slot high-water-mark. The same slot on multiple relays is normal: builders multi-home bids, so relay counters measure relay share, not a partition of blocks." - "Single vantage: one harness in one region observes the chain through one public RPC. Attribution is not latency-sensitive (extraData is consensus data, identical from every vantage), so single-vantage only affects liveness, which the health gauges expose." - "Companion L2 metrics: the same harness measures Arbitrum sequencer-feed soft-confirmation lag (ebb_arb_softconf_lag_milliseconds, feed arrival to public RPC visibility, offset derived at runtime) and Base flashblocks cadence plus flashblock-to-RPC lag (ebb_base_flashblock_interval_milliseconds, ebb_base_softconf_lag_milliseconds). These are per-chain, not per-builder, so this spec surfaces builder share only; the L2 lag series feed a future bench page." @@ -87,7 +87,7 @@ faq: - q: "Why do the p50, p90 and p99 columns show the same number?" a: "This bench has no latency dimension, so the percentile slots are repurposed, same convention as bench 016. Every aggregate column carries the builder's 24h block share; the sample column carries the raw block count and the success column reports the harness's own RPC poll health, not anything about the builder. The ranking signal is the share itself." - q: "What is the relay cross-check and why does it not sum to 100 percent?" - a: "Seven MEV-Boost relays expose a public data API listing every payload they delivered to a proposer. The harness counts delivered payloads per relay every five minutes. Because builders submit the same bid through several relays and every relay that carried the winning payload reports it, one block can legitimately appear on multiple relays. Relay counts therefore measure relay market share and act as an attribution sanity check independent of extraData; they are not a partition of blocks and do not sum to the block count." + a: "Six MEV-Boost relays expose a public data API listing every payload they delivered to a proposer. The harness counts delivered payloads per relay every five minutes. Because builders submit the same bid through several relays and every relay that carried the winning payload reports it, one block can legitimately appear on multiple relays. Relay counts therefore measure relay market share and act as an attribution sanity check independent of extraData; they are not a partition of blocks and do not sum to the block count." - q: "Why does builder concentration matter?" a: "Under MEV-Boost, validators outsource block construction, so whoever builds most blocks decides transaction ordering and inclusion for most of Ethereum. Concentration in two or three private firms creates censorship surface (a dominant builder filtering transactions affects most blocks), a single point of failure for liveness of the fee market, and information asymmetry for anyone trading on-chain. Projects like BuilderNet exist specifically to mutualize this layer; this bench provides the neutral number for whether concentration is getting better or worse." diff --git a/harnesses/evm-block-builders/cmd/script/relays.go b/harnesses/evm-block-builders/cmd/script/relays.go index 91c7f9226..a7ca75bf7 100644 --- a/harnesses/evm-block-builders/cmd/script/relays.go +++ b/harnesses/evm-block-builders/cmd/script/relays.go @@ -32,12 +32,19 @@ type relayEndpoint struct { } // relays verified answering keyless at bench inception (2026-07-14). +// +// bloXroute's max-profit relay was dropped on 2026-10-09: it has no DNS +// record at all (NXDOMAIN at 8.8.8.8, so not a local resolver artefact) and +// delivered 0 payloads in the preceding 24h while the other six all moved. +// Its counter kept a plausible-looking 26,395 total only because a Prometheus +// counter never goes down, which is exactly why the row looked alive. The +// spec advertised "seven MEV-Boost relays" in four places and has been +// corrected to six. func relays() []relayEndpoint { return []relayEndpoint{ {"flashbots", "https://boost-relay.flashbots.net"}, {"ultrasound", "https://relay.ultrasound.money"}, {"agnostic", "https://agnostic-relay.net"}, - {"bloxroute-max-profit", "https://bloxroute.max-profit.blxrbdn.com"}, {"bloxroute-regulated", "https://bloxroute.regulated.blxrbdn.com"}, {"titan", "https://titanrelay.xyz"}, {"aestus", "https://aestus.live"}, From 802f8da095809f0f80cac07d019e227cb3c12bd6 Mon Sep 17 00:00:00 2001 From: Flotapponnier <160007691+Flotapponnier@users.noreply.github.com> Date: Fri, 9 Oct 2026 15:05:39 +0200 Subject: [PATCH 2/2] bench 025 has been measuring three oracles while saying four (#2920) Pyth Hermes' price paths now answer 401 to an anonymous client, and the harness hardcodes one of them. Its Pyth leg has fetched nothing for at least 30 days: ocb_oracle_price{source="pyth"} shows 0 value changes over the full retention window, against 746,773 for coinbase. The published numbers were never wrong, which is exactly why nobody noticed. The headline reads the time-aligned family, and lookupNearest cannot find a 30-day-old point inside alignTolerance, so Pyth was dropped and the figures stayed honest 3-source deviations (6 to 8.5 bps, entirely plausible). What was wrong is everything the page said. The subtitle, the SEO description, the abstract, the Sources bullet and two FAQ answers all named Pyth as one of four live sources, and the Sources bullet called all four "free, no-auth, public endpoints". Corrected to three, with the reason stated. Detection failed three ways and all three are worth remembering: the price gauge holds its last value forever, ocb_oracle_alignment_miss_total reads 0 for every Pyth pair because a miss is only counted once a lookup is attempted, and the one counter that did move, ocb_oracle_scrape_errors_total at ~34,500 a day, has no reader. Same shape as the dead MEV relay found an hour earlier: a counter that never goes down makes a dead source look alive. The poller is commented out rather than deleted, so a key restores it in one line. It was also sending ~34,500 rejected requests a day to a third party. title and seo_title are deliberately untouched: they carry the page's ranking history on "Chainlink vs Pyth", and retitling, or paying for a key to restore the fourth source, is a call with a cost attached. go build and go vet clean, pnpm validate clean. --- benchmarks/oracle-deviation.yml | 24 ++++++++++--------- harnesses/oracle-deviation/cmd/script/main.go | 20 +++++++++++++++- 2 files changed, 32 insertions(+), 12 deletions(-) diff --git a/benchmarks/oracle-deviation.yml b/benchmarks/oracle-deviation.yml index 6f3ed8ddf..1f7fe2a01 100644 --- a/benchmarks/oracle-deviation.yml +++ b/benchmarks/oracle-deviation.yml @@ -4,8 +4,8 @@ slug: oracle-deviation number: "025" title: Chainlink vs Pyth vs Binance vs Coinbase, live oracle deviation seo_title: "Chainlink vs Pyth oracle deviation 2026" -seo_description: "Chainlink, Pyth, Binance, Coinbase oracle deviation ranked live on BTC, ETH, SOL, BNB, XRP, ADA, DOGE, AVAX, LINK, POL. Max pairwise bps, p99 over 24h." -subtitle: Maximum pairwise price disagreement in basis points across Chainlink, Pyth, Binance and Coinbase, polled every 30 seconds on 10 USD pairs. +seo_description: "Chainlink, Binance and Coinbase oracle deviation ranked live on BTC, ETH, SOL, BNB, XRP, ADA, DOGE, AVAX, LINK, POL. Max pairwise bps, p99 over 24h. Pyth paused since its endpoint went keyed." +subtitle: Maximum pairwise price disagreement in basis points across Chainlink, Binance and Coinbase, polled every 30 seconds on 10 USD pairs. Pyth is paused, its free price endpoint now requires a key. category: Trading status: live metric: Max cross-oracle deviation @@ -26,10 +26,12 @@ seo_intro: | `|a-b|/avg * 100` for every source pair, and publish the per-asset maximum as a Prometheus gauge in basis points. Coverage. Chainlink AggregatorV3 (eth_call against the canonical Ethereum mainnet - contracts), Pyth Hermes REST (batched latest_price_feeds), Binance - REST ticker (USDT-quoted, treated as ≈ USD), Coinbase REST ticker - (USD-quoted). Pairs. BTC, ETH, SOL, BNB, AVAX, LINK, POL (4 - oracles each) and XRP, ADA, DOGE (3 oracles, Chainlink mainnet + contracts), Binance REST ticker (USDT-quoted, treated as ≈ USD), + Coinbase REST ticker (USD-quoted). Pyth Hermes was the fourth source + until its price endpoints began answering 401 without a key; it has + contributed nothing since at least 2026-09-09 and is excluded rather + than averaged in stale. Pairs. BTC, ETH, SOL, BNB, AVAX, LINK, POL (3 + oracles each) and XRP, ADA, DOGE (2 oracles, Chainlink mainnet feeds for these pairs are deprecated, documented per provider). Honest about the SOL case. SOL/USD currently shows the widest disagreement (~1.2% live) not because the bench is broken but @@ -61,9 +63,9 @@ abstract: | feed vs. a CEX outlier) without confusing the reader. methodology: - - "Sources. Chainlink AggregatorV3 via `eth_call(latestRoundData())` on Ethereum mainnet (no API key, public RPC with fallback), Pyth Network via Hermes REST `/api/latest_price_feeds` batch endpoint (no key), Binance REST `/api/v3/ticker/price` on USDT-quoted symbols (treated as ≈ USD), Coinbase REST `/products/

/ticker` on USD-quoted products. All four are free, no-auth, public endpoints." - - "Pairs (10). BTC, ETH, SOL, BNB, AVAX, LINK and POL are tracked on all 4 sources. XRP, ADA, DOGE are tracked on 3 sources (Pyth + Binance + Coinbase) because the Chainlink mainnet AggregatorV3 contracts for these USD feeds are deprecated and no longer updated by node operators, the provider tag flags it." - - "Cadence. 30 second polling interval per (source, pair). 4 sources × 10 pairs × 2/minute = 80 requests/minute total, comfortably under every free-tier ceiling (Binance 1200/min, Coinbase 10/s public, Pyth ~30/s soft, public RPC courteous). HTTP timeout is 8 s per request to match the gas-estimation harness." + - "Sources. Chainlink AggregatorV3 via `eth_call(latestRoundData())` on Ethereum mainnet (no API key, public RPC with fallback), Binance REST `/api/v3/ticker/price` on USDT-quoted symbols (treated as ≈ USD), Coinbase REST `/products/

/ticker` on USD-quoted products. All three are free, no-auth, public endpoints. Pyth Hermes was a fourth until its price paths began returning 401 without a key." + - "Pairs (10). BTC, ETH, SOL, BNB, AVAX, LINK and POL are tracked on all 4 sources. XRP, ADA, DOGE are tracked on 2 sources (Binance + Coinbase) because the Chainlink mainnet AggregatorV3 contracts for these USD feeds are deprecated and no longer updated by node operators, the provider tag flags it." + - "Cadence. 30 second polling interval per (source, pair). 3 sources × 10 pairs × 2/minute = 60 requests/minute total, comfortably under every free-tier ceiling (Binance 1200/min, Coinbase 10/s public, public RPC courteous). HTTP timeout is 8 s per request to match the gas-estimation harness." - "Deviation formula. For every unordered pair of sources `(a, b)` that both have a fresh sample for the same pair: `deviation_pct = |price_a - price_b| / ((price_a + price_b) / 2) * 100`. With 4 sources we compute 6 pairwise deviations per asset; with 3 sources we compute 3. Per-pair headline is the max across all source-pairs (`ocb_oracle_max_deviation_pct`)." - "Stale-price guard. A source's sample is considered stale and excluded from the deviation calculation if its last successful update is older than `2 * pollInterval` (60 s). Prevents a dead Coinbase ticker from registering as a 0% deviation against itself by leaving the price gauge fossilized." - "Chainlink round age. The on-chain `updatedAt` timestamp from `latestRoundData()` is published separately as `ocb_oracle_last_round_age_seconds{source=\"chainlink\", pair}`. Chainlink updates only on deviation (typically 0.25-0.5% for blue chips) or heartbeat (~1 h), so a 30-minute age on a quiet ETH minute is normal; a 2-hour age on SOL during a volatile minute is the actual signal." @@ -75,8 +77,8 @@ methodology: findings: - "{{best_name}} currently leads at {{best_p50}} (p99 max cross-oracle deviation, 24 h) across {{count}} measured pairs. The leaderboard ranks by how tightly the available sources agree on the asset's price, lower = better, so the leader is the asset for which every source you might integrate would settle within fractions of a basis point of every other." - "{{name:btc-usd}} clocks {{p50:btc-usd}} on 4 sources. BTC's deep liquidity on every CEX plus the most-watched Chainlink feed on mainnet (heartbeats updated within minutes by competing node operators) means the 4-way disagreement is structurally near the floor; if you can't get oracle agreement on BTC, you have a poller bug, not a market signal." - - "{{name:sol-usd}} runs wider at {{p50:sol-usd}} on 4 sources. This is a real signal, not a bench bug. Chainlink SOL/USD updates only on 0.5% deviation or 1h heartbeat while Pyth/Binance/Coinbase publish continuously, during fast SOL moves the on-chain Chainlink price lags the CEX print by minutes (`ocb_oracle_last_round_age_seconds` often shows multi-thousand-second ages vs ~1000-1500s for blue chips). Any perp DEX settling SOL liquidations off Chainlink alone inherits this as trader slippage." - - "XRP, ADA and DOGE are measured on 3 sources, not 4, the Chainlink mainnet AggregatorV3 contracts for these USD pairs are deprecated and no longer updated by node operators, so the bench drops them from the deviation matrix rather than averaging in stale data. The 3-source deviation is honest (Pyth + Binance + Coinbase, all live), and the provider tag flags the asymmetry. Protocols settling these assets on Chainlink mainnet should confirm directly with the contract's `updatedAt`." + - "{{name:sol-usd}} runs wider at {{p50:sol-usd}}. This is a real signal, not a bench bug. Chainlink SOL/USD updates only on 0.5% deviation or 1h heartbeat while Binance and Coinbase publish continuously, during fast SOL moves the on-chain Chainlink price lags the CEX print by minutes (`ocb_oracle_last_round_age_seconds` often shows multi-thousand-second ages vs ~1000-1500s for blue chips). Any perp DEX settling SOL liquidations off Chainlink alone inherits this as trader slippage." + - "XRP, ADA and DOGE are measured on 2 sources, not 3, the Chainlink mainnet AggregatorV3 contracts for these USD pairs are deprecated and no longer updated by node operators, so the bench drops them from the deviation matrix rather than averaging in stale data. The 2-source deviation is honest (Binance + Coinbase, both live), and the provider tag flags the asymmetry. Protocols settling these assets on Chainlink mainnet should confirm directly with the contract's `updatedAt`." - "{{name:matic-usd}} sits at {{p50:matic-usd}}, the post-migration POL token. The Chainlink mainnet contract still answers to the legacy MATIC/USD name but on-chain `description()` confirms it tracks POL 1:1 (verified against Pyth's renamed POL/USD feed and Coinbase's POL-USD product). If you're building on Polygon and integrating the legacy contract address, you're already on POL, no migration needed." - "{{worst_name}} trails at {{worst_p50}} on the active filter. The gap between leader and laggard is dominated by two factors: how fast the asset moves (volatile minutes widen the deviation regardless of oracle quality) and how aggressive each oracle's update trigger is (Chainlink's deviation threshold is the binding constraint on most pairs; Pyth/Binance/Coinbase are within bps of each other most of the time)." diff --git a/harnesses/oracle-deviation/cmd/script/main.go b/harnesses/oracle-deviation/cmd/script/main.go index f3e9869c4..b13495410 100644 --- a/harnesses/oracle-deviation/cmd/script/main.go +++ b/harnesses/oracle-deviation/cmd/script/main.go @@ -351,7 +351,25 @@ func main() { defer cancel() go runChainlink(ctx, specs) - go runPyth(ctx, specs) + // Pyth is paused, not removed. Hermes' price paths + // (/api/latest_price_feeds and /v2/updates/price/latest) now answer 401 + // to an anonymous client, so this poller had fetched nothing for at + // least 30 days while burning ~34,500 rejected requests a day at a third + // party. The gate is endpoint-scoped, not an IP block: /v2/price_feeds + // on the same host still returns 200. + // + // Nothing published was wrong, which is why it went unnoticed. The + // headline reads the time-aligned family, and lookupNearest cannot find + // a 30-day-old point inside alignTolerance, so Pyth was silently dropped + // and the figures stayed honest 3-source deviations. What failed is + // detection: the price gauge holds its last value forever, and + // ocb_oracle_alignment_miss_total reads 0 for every Pyth pair because a + // miss is only counted once a lookup is attempted. Only + // ocb_oracle_scrape_errors_total moved, and nobody reads it. + // + // Re-enable the moment a key exists: set PYTH_HERMES_KEY and restore the + // call. See the spec's Sources bullet, which now says three. + // go runPyth(ctx, specs) go runBinance(ctx, specs) go runCoinbase(ctx, specs) go runLatencyUpdater(ctx)