From f5895d1ca42dfb5dab49128849fe92dc69628b14 Mon Sep 17 00:00:00 2001 From: Flotapponnier <160007691+Flotapponnier@users.noreply.github.com> Date: Wed, 7 Oct 2026 16:17:27 +0200 Subject: [PATCH] bench 284 called them frontier models without saying frontier of when Florent's instinct was right and it is worse than it looked. The field was enrolled on 2025-12-24 and 2026-01-14 and has not been refreshed since: grok 4 vision, gemini 3 pro vision 2025-12-24 sonnet 4.5 vision, gpt-5.2 vision 2025-12-24 gpt-5.2 chart, opus 4.5 chart 2026-01-14 grok 4 chart, gemini 3 pro chart 2026-01-14 266 days. The page said "eight frontier models" with no date on it, which is a claim about the frontier at enrolment presented as a claim about now. Opus 4.5 and Sonnet 4.5 are a generation behind the current Claude family. We cannot fix it by widening the field. The arena is allowlistOnly with maxParticipants 8, so Recall picks the roster and we measure what it runs. What we can do is date the claim, which is what this does. ## Shipped trading_agent_registered_timestamp_seconds per agent, and trading_agent_roster_age_days at arena level: days since the MOST RECENT enrolment, so one agent added tomorrow would correctly read as a refreshed roster rather than being hidden behind the oldest date. The hub states it above the table, from the live figure, and only when the gap is wide enough to matter. The spec's seo_intro, methodology and the "which model is best" FAQ all carry it now. The spec was at 16 metric panels of a maximum 17, so rather than sit at the ceiling this drops the trades_total panel, which was data-only, backed no column, and is the one a reader would miss least. The metric stays in the harness and remains queryable in Prometheus. ## The check that keeps the copy honest Plugin R11 reads each agent's enrolment date from a fresh upstream walk and WARNS when the roster is newer than the copy claims, naming the four places that then have to move. Same shape as R8, which warns if a pair of agents ever starts to separate. Without it the dated sentence would silently become false the day Recall enrols someone, because this bench scores rounds and would otherwise never look at the roster. ## Something I got wrong and corrected While checking whether Recall had newer models registered anywhere, I sampled 300 agents and was about to report that there were none. The registry holds 15,485. A 2 % sample supports no such claim, and I only found out because the API rate-limited me into reading the pagination total. The finding above rests only on the eight agents' own records, which are verified. --- benchmarks/trading-agent-alpha.yml | 14 ++++---- harnesses/trading-agent-alpha/activity.go | 16 ++++++++- harnesses/trading-agent-alpha/main.go | 42 ++++++++++++++++++++--- src/app/trading-agents/page.tsx | 14 +++++++- src/lib/trading-agent-hub.ts | 5 +++ 5 files changed, 79 insertions(+), 12 deletions(-) diff --git a/benchmarks/trading-agent-alpha.yml b/benchmarks/trading-agent-alpha.yml index dc1e57835..1c522f6cf 100644 --- a/benchmarks/trading-agent-alpha.yml +++ b/benchmarks/trading-agent-alpha.yml @@ -15,7 +15,7 @@ higher_is_better: true row_noun: { one: agent, many: agents } seo_intro: | - Recall Labs runs a weekly spot-trading arena on Base where eight autonomous agents trade the same ETH/USDC pair for six days under identical rules. Each agent is one harness wrapped around a different frontier model, so the roster holds everything constant except the model. Recall publishes who won each round. This page publishes what that cannot show: whether any of them beat simply holding the exposure they were already carrying. + Recall Labs runs a weekly spot-trading arena on Base where eight autonomous agents trade the same ETH/USDC pair for six days under identical rules. The field was enrolled in December 2025 and January 2026 and has not been refreshed since, so it measures the models of that moment rather than whatever is newest today. Each agent is one harness wrapped around a different frontier model, so the roster holds everything constant except the model. Recall publishes who won each round. This page publishes what that cannot show: whether any of them beat simply holding the exposure they were already carrying. abstract: | Eight frontier models, same token, same rules, {{count}} ranked over the weekly rounds each one funded. {{best_name}} leads at {{best_p50}}; the trailer is {{worst_name}} at {{worst_p50}}. Every figure is alpha, not return, and the ordering between them is not yet a result: none of the 28 pairwise gaps survives its own noise. @@ -29,6 +29,7 @@ methodology: - "Beta is measured per agent, not assumed, by regressing its round returns on the asset's return over the same windows. The measured betas run from 0.12 to 0.74, so a single blanket exposure assumption would flatter some agents and penalise others." - "A round with a zero profit figure is almost never a flat trade. Of the zero rows in this arena, 88 of 90 carry a portfolio value of zero, meaning the agent did not fund that round. Only funded rounds count. Treating them as zero-percent results drops the measured hit rate by eleven points and inflates each sample from about 22 rounds to 34." - "Eleven ended rounds publish ranks from 1 to 8 and no data at all, every portfolio value zero. The harness refuses to score a round without a real field, and publishes the count it refused as `trading_agent_rounds_skipped`." + - "Roster age, published rather than implied: these eight agents were enrolled upstream on 2025-12-24 and 2026-01-14 and the field has not changed since. `trading_agent_roster_age_days` carries the gap so the page never reads as a statement about the current frontier. We cannot widen it either: the arena is allowlisted at eight places and Recall picks them." - "The roster changed once. Only agents with at least ten funded rounds are ranked, which selects the eight-model roster that 32 rounds share. A handful of other entries appear two or three times and are not averaged into the same table." - "The asset counterfactual comes from Coinbase's public ETH-USD daily closes, deliberately outside Recall, because a benchmark drawn from the same source as the thing it measures is circular. Where a round boundary has no candle the harness walks back up to five days, and a round with no price is skipped rather than scored against nothing." - "What this does not measure: Recall builds the agents. Their own description reads `Autonomous live spot trading agent built by Recall Labs, powered by` the model in question, so the prompt, the rebalancing and the execution are Recall's and are held constant. This is one harness interacting with each model, never a model's trading ability in the abstract." @@ -58,7 +59,7 @@ faq: - q: "Is the gap between these agents meaningful?" a: "No, and the page says so rather than letting the ordering imply otherwise. Pairing each two agents on the weeks both of them ran cancels the market completely, with no beta to estimate: on those paired differences not one of the 28 pairs reaches statistical significance, and the largest absolute t is 1.36. Read the table as an order of finish over the rounds so far." - q: "Which AI model is the best trader?" - a: "This data cannot answer that, and a page that claimed to would be overstating it. Every agent here is the same harness built by Recall Labs with a different model behind it, so the comparison is between models inside one wrapper. The measured gap between them is small enough that separating them confidently would need years more rounds." + a: "This data cannot answer that, and a page that claimed to would be overstating it. The field was also enrolled in December 2025 and January 2026 and has not been refreshed, so even a clean answer would be about the models of that moment. Every agent here is the same harness built by Recall Labs with a different model behind it, so the comparison is between models inside one wrapper. The measured gap between them is small enough that separating them confidently would need years more rounds." - q: "Why measure alpha instead of return?" a: "The arena is single-asset: every agent holds a mix of ETH and USDC, so a round's return is mostly ETH's move. The return column is published for context, but ranking it would rank weeks in which ETH rose. Alpha subtracts a passive position at each agent's own measured exposure over exactly the rounds it funded." - q: "Are they losing to trading costs rather than bad decisions?" @@ -190,13 +191,14 @@ metric_panels: metric: trading_agent_trades_per_round{kind="all"} unit: count description: "How often this agent trades in a six-day round. Under identical rules the roster spans 2 to 89, a forty-fold difference in how much the same harness acts depending only on which model drives it." - - id: trades_total - label: Trades, lifetime + + - id: roster_age + label: Days since the roster was refreshed label_key: agent - metric: trading_agent_trades_total{kind="all"} + metric: trading_agent_roster_age_days{kind="all"} unit: count tab: false - description: "Lifetime trades the upstream reports. Checked against the agent's own round list: 35 to 37 of each agent's rounds are this arena, so the counter is not diluted by others." + description: "ARENA-LEVEL, identical on every row. Days since the most recent agent was enrolled. The field went in once and has not changed since, so calling these frontier models dates that claim rather than making it about today." ledger_columns: - { label: "Alpha", slot: p50 } diff --git a/harnesses/trading-agent-alpha/activity.go b/harnesses/trading-agent-alpha/activity.go index b1e06b941..d57d5b4c1 100644 --- a/harnesses/trading-agent-alpha/activity.go +++ b/harnesses/trading-agent-alpha/activity.go @@ -3,6 +3,7 @@ package main import ( "fmt" "math" + "time" ) // How often each agent actually trades. @@ -44,6 +45,7 @@ type agentDetail struct { Agent struct { Name string `json:"name"` WalletAddress string `json:"walletAddress"` + CreatedAt string `json:"createdAt"` Stats struct { TotalTrades *float64 `json:"totalTrades"` CompletedCompetitions *float64 `json:"completedCompetitions"` @@ -56,6 +58,11 @@ type activity struct { totalTrades float64 tradesPerRound float64 wallet string + // registered is when the agent was created upstream. The roster was + // enrolled once, in December 2025 and January 2026, and never refreshed, + // so "frontier models" is a claim about the frontier AT ENROLMENT. The + // page has no business making it undated. + registered time.Time } // fetchActivity reads one record per agent id. @@ -83,11 +90,18 @@ func fetchActivity(ids map[string]string) map[string]activity { if math.IsNaN(per) || math.IsInf(per, 0) { continue } - out[name] = activity{ + act := activity{ totalTrades: *st.TotalTrades, tradesPerRound: per, wallet: d.Agent.WalletAddress, } + // RFC3339 with milliseconds, e.g. 2026-01-14T06:28:12.853Z. A date we + // cannot parse is left zero and simply not published, rather than + // defaulting to now, which would read as a roster refreshed today. + if t, err := time.Parse(time.RFC3339, d.Agent.CreatedAt); err == nil { + act.registered = t + } + out[name] = act } return out } diff --git a/harnesses/trading-agent-alpha/main.go b/harnesses/trading-agent-alpha/main.go index 6c5bf378d..30761a474 100644 --- a/harnesses/trading-agent-alpha/main.go +++ b/harnesses/trading-agent-alpha/main.go @@ -196,6 +196,16 @@ var ( Help: "Lifetime trades the upstream reports for this agent. Checked against /agents/{id}/competitions: 35 to 37 of each agent's rounds are this arena, so the counter is not diluted by others.", }, []string{"agent", "kind", "arena"}) + registeredAt = prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "trading_agent_registered_timestamp_seconds", + Help: "When this agent was enrolled upstream. The roster went in once, December 2025 and January 2026, and has not been refreshed since.", + }, []string{"agent", "kind", "arena"}) + + rosterAgeDays = prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "trading_agent_roster_age_days", + Help: "ARENA-LEVEL. Days since the MOST RECENT enrolment on the roster, i.e. how long since the field was last refreshed. A benchmark calling these agents frontier models is dating that claim, not making it about today.", + }, []string{"agent", "kind", "arena"}) + roundsScored = prometheus.NewGaugeVec(prometheus.GaugeOpts{ Name: "trading_agent_rounds_scored", Help: "Rounds in the arena that carried a real field and were scored.", @@ -227,7 +237,7 @@ func init() { alphaPct, returnPct, passivePct, assetPct, hitRatePct, beta, rounds, weeklySD, alphaT, pooledAlphaPct, pooledAlphaT, roundsNeeded, separableCount, scheduledRounds, daysSinceRound, - tradesPerRound, tradesTotal, + tradesPerRound, tradesTotal, registeredAt, rosterAgeDays, roundsScored, roundsSkipped, lastRun, lastRoundEnd, errors, } { prometheus.MustRegister(c) @@ -517,6 +527,8 @@ func runOnce() error { daysSinceRound.Reset() tradesPerRound.Reset() tradesTotal.Reset() + registeredAt.Reset() + rosterAgeDays.Reset() roundsSkipped.Reset() // Betas are needed by the pooled test, which cannot run until every @@ -646,14 +658,36 @@ func runOnce() error { wanted[name] = id } } - for name, act := range fetchActivity(wanted) { + acts := fetchActivity(wanted) + // The newest enrolment, which is how long since the field was refreshed. + // Newest rather than oldest: one agent added yesterday would make the + // roster current, and the oldest date would hide that. + var newest time.Time + for name, act := range acts { for _, k := range []string{kindOf(name), "all"} { lbl := []string{slugOf(name), k, arena} tradesPerRound.WithLabelValues(lbl...).Set(act.tradesPerRound) tradesTotal.WithLabelValues(lbl...).Set(act.totalTrades) + if !act.registered.IsZero() { + registeredAt.WithLabelValues(lbl...).Set(float64(act.registered.Unix())) + } + } + if act.registered.After(newest) { + newest = act.registered + } + log.Printf("[284] %-22s %7.0f trades, %6.2f per round, enrolled %s, wallet %s", + name, act.totalTrades, act.tradesPerRound, + act.registered.Format("2006-01-02"), act.wallet) + } + if !newest.IsZero() { + age := time.Since(newest).Hours() / 24 + for _, name := range ranked { + for _, k := range []string{kindOf(name), "all"} { + rosterAgeDays.WithLabelValues(slugOf(name), k, arena).Set(age) + } } - log.Printf("[284] %-22s %7.0f trades, %6.2f per round, wallet %s", - name, act.totalTrades, act.tradesPerRound, act.wallet) + log.Printf("[284] roster: last enrolled %s, %.0f days ago", + newest.Format("2006-01-02"), age) } // Liveness, published per agent for the same reason the arena-level diff --git a/src/app/trading-agents/page.tsx b/src/app/trading-agents/page.tsx index d49dd2675..34e63ad21 100644 --- a/src/app/trading-agents/page.tsx +++ b/src/app/trading-agents/page.tsx @@ -87,6 +87,18 @@ export default async function TradingAgentsHubPage() { show: whether any of them beat holding the exposure they were already carrying.

+ {hub?.arena.rosterAgeDays != null && hub.arena.rosterAgeDays > 120 && ( +

+ One caveat before the numbers: the field was enrolled{" "} + + {fmtInt(hub.arena.rosterAgeDays)} days ago + {" "} + and has not been refreshed, so these are the frontier models of + that moment rather than whatever is newest today. The arena is + allowlisted at eight places and Recall chooses them, so the page + dates the claim instead of quietly making it about the present. +

+ )}
{ arena: { scheduledRounds: arenaStat("scheduled_rounds"), daysSinceLastRound: arenaStat("days_since_round"), + rosterAgeDays: arenaStat("roster_age"), }, }; }