Repository navigation
bench 284 rendered a leaderboard while its own copy said not to read one - #2850
Merged
Merged
Conversation
The page told the reader "an order of finish, not a ranking" and then drew
eight rows sorted by alpha with a bold first place. Form beats text: nobody
reaches the paragraph. Zero of the 28 pairwise comparisons clears significance
on the weeks both agents ran, so the honest rendering shows them tied, and now
it does.
- a banner above the table states the tie, from the live figure
- labs are ordered ALPHABETICALLY; sorting them by best alpha was itself a
ranking claim
- the alpha cell loses its bold, which read as "the one to compare on"
- the "Best alpha" card becomes "Alpha range", a fact rather than a podium
## Found while checking the boost mechanic: the return is not the balance
Chasing whether boosting touches scoring, I pulled portfolio values per round
and they did not reconcile with what we publish.
agent published return wallet over the same rounds
gemini 3 pro vision -15.72 % 264.87 -> 3.33 (-98.7 %)
sonnet 4.5 vision -25.22 % 229.40 -> 10.74 (-95.3 %)
opus 4.5 chart -19.22 % 297.64 -> 302.76 (+1.7 %)
The last line is the one that matters. We publish -19.22 % for an agent whose
wallet finished UP.
Our arithmetic is not wrong. Compounding each round's percentage is a
time-weighted return, which deliberately ignores money moved in or out between
rounds because the arena decides those and the agent does not. That is the
standard way to judge a manager whose cash flows it does not control, and it
is the right choice here.
But the page never said which of the two it was showing, and a reader assuming
the Return column was the money would have been wrong by twenty points. It
says so now, in the methodology, in a new FAQ entry and under the table.
## Capital is small and did not hold still
Median portfolio is a few hundred dollars an agent, which the page already
hinted at and now states as a column. The new finding is the instability: the
largest-to-smallest ratio across an agent's own rounds runs from 1.3 to about
190, and half the roster is above 5.
That is not cosmetic. Gas is a fixed cost per swap, so an agent trading the
same way on three dollars carries drag an agent on three hundred does not, and
for those agents the late rounds are not the same experiment as the early
ones. Published as trading_agent_capital_ratio.
I had written "two agents near 86" from an earlier sample that only looked at
final values; the full walk gives 1.3 to 190 with grok 4 chart the widest. The
copy carries the measured range.
## The boost mechanic, which started all this
No evidence it touches scoring: boosterAllocation is null, rewardsTge is
{agentPool: 0, userPool: 1e22} so the rewards go to the humans who boost
rather than to agents, and /competitions/{id}/boosts is 401. Not provable from
outside, but nothing points the other way and the capital walk shows balances
moving only through trading and withdrawals.
## Housekeeping
The spec is now at 17 metric panels of a maximum 17 and 9 ledger columns of a
maximum 9. Both at the ceiling. `own_t` was dropped to make room: data-only,
backed no column, and the checks read that metric straight from Prometheus
rather than through the panel. Anything further needs something removed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bench 284 rendered a leaderboard while its own copy said not to read one
The page told the reader "an order of finish, not a ranking" and then drew
eight rows sorted by alpha with a bold first place. Form beats text: nobody
reaches the paragraph. Zero of the 28 pairwise comparisons clears significance
on the weeks both agents ran, so the honest rendering shows them tied, and now
it does.
ranking claim
Found while checking the boost mechanic: the return is not the balance
Chasing whether boosting touches scoring, I pulled portfolio values per round
and they did not reconcile with what we publish.
The last line is the one that matters. We publish -19.22 % for an agent whose
wallet finished UP.
Our arithmetic is not wrong. Compounding each round's percentage is a
time-weighted return, which deliberately ignores money moved in or out between
rounds because the arena decides those and the agent does not. That is the
standard way to judge a manager whose cash flows it does not control, and it
is the right choice here.
But the page never said which of the two it was showing, and a reader assuming
the Return column was the money would have been wrong by twenty points. It
says so now, in the methodology, in a new FAQ entry and under the table.
Capital is small and did not hold still
Median portfolio is a few hundred dollars an agent, which the page already
hinted at and now states as a column. The new finding is the instability: the
largest-to-smallest ratio across an agent's own rounds runs from 1.3 to about
190, and half the roster is above 5.
That is not cosmetic. Gas is a fixed cost per swap, so an agent trading the
same way on three dollars carries drag an agent on three hundred does not, and
for those agents the late rounds are not the same experiment as the early
ones. Published as trading_agent_capital_ratio.
I had written "two agents near 86" from an earlier sample that only looked at
final values; the full walk gives 1.3 to 190 with grok 4 chart the widest. The
copy carries the measured range.
The boost mechanic, which started all this
No evidence it touches scoring: boosterAllocation is null, rewardsTge is
{agentPool: 0, userPool: 1e22} so the rewards go to the humans who boost
rather than to agents, and /competitions/{id}/boosts is 401. Not provable from
outside, but nothing points the other way and the capital walk shows balances
moving only through trading and withdrawals.
Housekeeping
The spec is now at 17 metric panels of a maximum 17 and 9 ledger columns of a
maximum 9. Both at the ceiling.
own_twas dropped to make room: data-only,backed no column, and the checks read that metric straight from Prometheus
rather than through the panel. Anything further needs something removed.