diff --git a/benchmarks/trading-agent-alpha.yml b/benchmarks/trading-agent-alpha.yml index dc1e57835..1c522f6cf 100644 --- a/benchmarks/trading-agent-alpha.yml +++ b/benchmarks/trading-agent-alpha.yml @@ -15,7 +15,7 @@ higher_is_better: true row_noun: { one: agent, many: agents } seo_intro: | - Recall Labs runs a weekly spot-trading arena on Base where eight autonomous agents trade the same ETH/USDC pair for six days under identical rules. Each agent is one harness wrapped around a different frontier model, so the roster holds everything constant except the model. Recall publishes who won each round. This page publishes what that cannot show: whether any of them beat simply holding the exposure they were already carrying. + Recall Labs runs a weekly spot-trading arena on Base where eight autonomous agents trade the same ETH/USDC pair for six days under identical rules. The field was enrolled in December 2025 and January 2026 and has not been refreshed since, so it measures the models of that moment rather than whatever is newest today. Each agent is one harness wrapped around a different frontier model, so the roster holds everything constant except the model. Recall publishes who won each round. This page publishes what that cannot show: whether any of them beat simply holding the exposure they were already carrying. abstract: | Eight frontier models, same token, same rules, {{count}} ranked over the weekly rounds each one funded. {{best_name}} leads at {{best_p50}}; the trailer is {{worst_name}} at {{worst_p50}}. Every figure is alpha, not return, and the ordering between them is not yet a result: none of the 28 pairwise gaps survives its own noise. @@ -29,6 +29,7 @@ methodology: - "Beta is measured per agent, not assumed, by regressing its round returns on the asset's return over the same windows. The measured betas run from 0.12 to 0.74, so a single blanket exposure assumption would flatter some agents and penalise others." - "A round with a zero profit figure is almost never a flat trade. Of the zero rows in this arena, 88 of 90 carry a portfolio value of zero, meaning the agent did not fund that round. Only funded rounds count. Treating them as zero-percent results drops the measured hit rate by eleven points and inflates each sample from about 22 rounds to 34." - "Eleven ended rounds publish ranks from 1 to 8 and no data at all, every portfolio value zero. The harness refuses to score a round without a real field, and publishes the count it refused as `trading_agent_rounds_skipped`." + - "Roster age, published rather than implied: these eight agents were enrolled upstream on 2025-12-24 and 2026-01-14 and the field has not changed since. `trading_agent_roster_age_days` carries the gap so the page never reads as a statement about the current frontier. We cannot widen it either: the arena is allowlisted at eight places and Recall picks them." - "The roster changed once. Only agents with at least ten funded rounds are ranked, which selects the eight-model roster that 32 rounds share. A handful of other entries appear two or three times and are not averaged into the same table." - "The asset counterfactual comes from Coinbase's public ETH-USD daily closes, deliberately outside Recall, because a benchmark drawn from the same source as the thing it measures is circular. Where a round boundary has no candle the harness walks back up to five days, and a round with no price is skipped rather than scored against nothing." - "What this does not measure: Recall builds the agents. Their own description reads `Autonomous live spot trading agent built by Recall Labs, powered by` the model in question, so the prompt, the rebalancing and the execution are Recall's and are held constant. This is one harness interacting with each model, never a model's trading ability in the abstract." @@ -58,7 +59,7 @@ faq: - q: "Is the gap between these agents meaningful?" a: "No, and the page says so rather than letting the ordering imply otherwise. Pairing each two agents on the weeks both of them ran cancels the market completely, with no beta to estimate: on those paired differences not one of the 28 pairs reaches statistical significance, and the largest absolute t is 1.36. Read the table as an order of finish over the rounds so far." - q: "Which AI model is the best trader?" - a: "This data cannot answer that, and a page that claimed to would be overstating it. Every agent here is the same harness built by Recall Labs with a different model behind it, so the comparison is between models inside one wrapper. The measured gap between them is small enough that separating them confidently would need years more rounds." + a: "This data cannot answer that, and a page that claimed to would be overstating it. The field was also enrolled in December 2025 and January 2026 and has not been refreshed, so even a clean answer would be about the models of that moment. Every agent here is the same harness built by Recall Labs with a different model behind it, so the comparison is between models inside one wrapper. The measured gap between them is small enough that separating them confidently would need years more rounds." - q: "Why measure alpha instead of return?" a: "The arena is single-asset: every agent holds a mix of ETH and USDC, so a round's return is mostly ETH's move. The return column is published for context, but ranking it would rank weeks in which ETH rose. Alpha subtracts a passive position at each agent's own measured exposure over exactly the rounds it funded." - q: "Are they losing to trading costs rather than bad decisions?" @@ -190,13 +191,14 @@ metric_panels: metric: trading_agent_trades_per_round{kind="all"} unit: count description: "How often this agent trades in a six-day round. Under identical rules the roster spans 2 to 89, a forty-fold difference in how much the same harness acts depending only on which model drives it." - - id: trades_total - label: Trades, lifetime + + - id: roster_age + label: Days since the roster was refreshed label_key: agent - metric: trading_agent_trades_total{kind="all"} + metric: trading_agent_roster_age_days{kind="all"} unit: count tab: false - description: "Lifetime trades the upstream reports. Checked against the agent's own round list: 35 to 37 of each agent's rounds are this arena, so the counter is not diluted by others." + description: "ARENA-LEVEL, identical on every row. Days since the most recent agent was enrolled. The field went in once and has not changed since, so calling these frontier models dates that claim rather than making it about today." ledger_columns: - { label: "Alpha", slot: p50 } diff --git a/harnesses/trading-agent-alpha/activity.go b/harnesses/trading-agent-alpha/activity.go index b1e06b941..d57d5b4c1 100644 --- a/harnesses/trading-agent-alpha/activity.go +++ b/harnesses/trading-agent-alpha/activity.go @@ -3,6 +3,7 @@ package main import ( "fmt" "math" + "time" ) // How often each agent actually trades. @@ -44,6 +45,7 @@ type agentDetail struct { Agent struct { Name string `json:"name"` WalletAddress string `json:"walletAddress"` + CreatedAt string `json:"createdAt"` Stats struct { TotalTrades *float64 `json:"totalTrades"` CompletedCompetitions *float64 `json:"completedCompetitions"` @@ -56,6 +58,11 @@ type activity struct { totalTrades float64 tradesPerRound float64 wallet string + // registered is when the agent was created upstream. The roster was + // enrolled once, in December 2025 and January 2026, and never refreshed, + // so "frontier models" is a claim about the frontier AT ENROLMENT. The + // page has no business making it undated. + registered time.Time } // fetchActivity reads one record per agent id. @@ -83,11 +90,18 @@ func fetchActivity(ids map[string]string) map[string]activity { if math.IsNaN(per) || math.IsInf(per, 0) { continue } - out[name] = activity{ + act := activity{ totalTrades: *st.TotalTrades, tradesPerRound: per, wallet: d.Agent.WalletAddress, } + // RFC3339 with milliseconds, e.g. 2026-01-14T06:28:12.853Z. A date we + // cannot parse is left zero and simply not published, rather than + // defaulting to now, which would read as a roster refreshed today. + if t, err := time.Parse(time.RFC3339, d.Agent.CreatedAt); err == nil { + act.registered = t + } + out[name] = act } return out } diff --git a/harnesses/trading-agent-alpha/main.go b/harnesses/trading-agent-alpha/main.go index 6c5bf378d..30761a474 100644 --- a/harnesses/trading-agent-alpha/main.go +++ b/harnesses/trading-agent-alpha/main.go @@ -196,6 +196,16 @@ var ( Help: "Lifetime trades the upstream reports for this agent. Checked against /agents/{id}/competitions: 35 to 37 of each agent's rounds are this arena, so the counter is not diluted by others.", }, []string{"agent", "kind", "arena"}) + registeredAt = prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "trading_agent_registered_timestamp_seconds", + Help: "When this agent was enrolled upstream. The roster went in once, December 2025 and January 2026, and has not been refreshed since.", + }, []string{"agent", "kind", "arena"}) + + rosterAgeDays = prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "trading_agent_roster_age_days", + Help: "ARENA-LEVEL. Days since the MOST RECENT enrolment on the roster, i.e. how long since the field was last refreshed. A benchmark calling these agents frontier models is dating that claim, not making it about today.", + }, []string{"agent", "kind", "arena"}) + roundsScored = prometheus.NewGaugeVec(prometheus.GaugeOpts{ Name: "trading_agent_rounds_scored", Help: "Rounds in the arena that carried a real field and were scored.", @@ -227,7 +237,7 @@ func init() { alphaPct, returnPct, passivePct, assetPct, hitRatePct, beta, rounds, weeklySD, alphaT, pooledAlphaPct, pooledAlphaT, roundsNeeded, separableCount, scheduledRounds, daysSinceRound, - tradesPerRound, tradesTotal, + tradesPerRound, tradesTotal, registeredAt, rosterAgeDays, roundsScored, roundsSkipped, lastRun, lastRoundEnd, errors, } { prometheus.MustRegister(c) @@ -517,6 +527,8 @@ func runOnce() error { daysSinceRound.Reset() tradesPerRound.Reset() tradesTotal.Reset() + registeredAt.Reset() + rosterAgeDays.Reset() roundsSkipped.Reset() // Betas are needed by the pooled test, which cannot run until every @@ -646,14 +658,36 @@ func runOnce() error { wanted[name] = id } } - for name, act := range fetchActivity(wanted) { + acts := fetchActivity(wanted) + // The newest enrolment, which is how long since the field was refreshed. + // Newest rather than oldest: one agent added yesterday would make the + // roster current, and the oldest date would hide that. + var newest time.Time + for name, act := range acts { for _, k := range []string{kindOf(name), "all"} { lbl := []string{slugOf(name), k, arena} tradesPerRound.WithLabelValues(lbl...).Set(act.tradesPerRound) tradesTotal.WithLabelValues(lbl...).Set(act.totalTrades) + if !act.registered.IsZero() { + registeredAt.WithLabelValues(lbl...).Set(float64(act.registered.Unix())) + } + } + if act.registered.After(newest) { + newest = act.registered + } + log.Printf("[284] %-22s %7.0f trades, %6.2f per round, enrolled %s, wallet %s", + name, act.totalTrades, act.tradesPerRound, + act.registered.Format("2006-01-02"), act.wallet) + } + if !newest.IsZero() { + age := time.Since(newest).Hours() / 24 + for _, name := range ranked { + for _, k := range []string{kindOf(name), "all"} { + rosterAgeDays.WithLabelValues(slugOf(name), k, arena).Set(age) + } } - log.Printf("[284] %-22s %7.0f trades, %6.2f per round, wallet %s", - name, act.totalTrades, act.tradesPerRound, act.wallet) + log.Printf("[284] roster: last enrolled %s, %.0f days ago", + newest.Format("2006-01-02"), age) } // Liveness, published per agent for the same reason the arena-level diff --git a/src/app/trading-agents/page.tsx b/src/app/trading-agents/page.tsx index d49dd2675..34e63ad21 100644 --- a/src/app/trading-agents/page.tsx +++ b/src/app/trading-agents/page.tsx @@ -87,6 +87,18 @@ export default async function TradingAgentsHubPage() { show: whether any of them beat holding the exposure they were already carrying.

+ {hub?.arena.rosterAgeDays != null && hub.arena.rosterAgeDays > 120 && ( +

+ One caveat before the numbers: the field was enrolled{" "} + + {fmtInt(hub.arena.rosterAgeDays)} days ago + {" "} + and has not been refreshed, so these are the frontier models of + that moment rather than whatever is newest today. The arena is + allowlisted at eight places and Recall chooses them, so the page + dates the claim instead of quietly making it about the present. +

+ )}
{ arena: { scheduledRounds: arenaStat("scheduled_rounds"), daysSinceLastRound: arenaStat("days_since_round"), + rosterAgeDays: arenaStat("roster_age"), }, }; }