Skip to content

bench 284 rendered a leaderboard while its own copy said not to read one - #2850

Merged
Flotapponnier merged 1 commit into
devfrom
feat/284-tied-not-ranked
Oct 7, 2026
Merged

Flotapponnier merged 1 commit into
devfrom
feat/284-tied-not-ranked

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

bench 284 rendered a leaderboard while its own copy said not to read one

The page told the reader "an order of finish, not a ranking" and then drew
eight rows sorted by alpha with a bold first place. Form beats text: nobody
reaches the paragraph. Zero of the 28 pairwise comparisons clears significance
on the weeks both agents ran, so the honest rendering shows them tied, and now
it does.

  • a banner above the table states the tie, from the live figure
  • labs are ordered ALPHABETICALLY; sorting them by best alpha was itself a
    ranking claim
  • the alpha cell loses its bold, which read as "the one to compare on"
  • the "Best alpha" card becomes "Alpha range", a fact rather than a podium

Found while checking the boost mechanic: the return is not the balance

Chasing whether boosting touches scoring, I pulled portfolio values per round
and they did not reconcile with what we publish.

agent                  published return    wallet over the same rounds
gemini 3 pro vision          -15.72 %      264.87 -> 3.33     (-98.7 %)
sonnet 4.5 vision            -25.22 %      229.40 -> 10.74    (-95.3 %)
opus 4.5 chart               -19.22 %      297.64 -> 302.76   (+1.7 %)

The last line is the one that matters. We publish -19.22 % for an agent whose
wallet finished UP.

Our arithmetic is not wrong. Compounding each round's percentage is a
time-weighted return, which deliberately ignores money moved in or out between
rounds because the arena decides those and the agent does not. That is the
standard way to judge a manager whose cash flows it does not control, and it
is the right choice here.

But the page never said which of the two it was showing, and a reader assuming
the Return column was the money would have been wrong by twenty points. It
says so now, in the methodology, in a new FAQ entry and under the table.

Capital is small and did not hold still

Median portfolio is a few hundred dollars an agent, which the page already
hinted at and now states as a column. The new finding is the instability: the
largest-to-smallest ratio across an agent's own rounds runs from 1.3 to about
190, and half the roster is above 5.

That is not cosmetic. Gas is a fixed cost per swap, so an agent trading the
same way on three dollars carries drag an agent on three hundred does not, and
for those agents the late rounds are not the same experiment as the early
ones. Published as trading_agent_capital_ratio.

I had written "two agents near 86" from an earlier sample that only looked at
final values; the full walk gives 1.3 to 190 with grok 4 chart the widest. The
copy carries the measured range.

The boost mechanic, which started all this

No evidence it touches scoring: boosterAllocation is null, rewardsTge is
{agentPool: 0, userPool: 1e22} so the rewards go to the humans who boost
rather than to agents, and /competitions/{id}/boosts is 401. Not provable from
outside, but nothing points the other way and the capital walk shows balances
moving only through trading and withdrawals.

Housekeeping

The spec is now at 17 metric panels of a maximum 17 and 9 ledger columns of a
maximum 9. Both at the ceiling. own_t was dropped to make room: data-only,
backed no column, and the checks read that metric straight from Prometheus
rather than through the panel. Anything further needs something removed.

The page told the reader "an order of finish, not a ranking" and then drew
eight rows sorted by alpha with a bold first place. Form beats text: nobody
reaches the paragraph. Zero of the 28 pairwise comparisons clears significance
on the weeks both agents ran, so the honest rendering shows them tied, and now
it does.

  - a banner above the table states the tie, from the live figure
  - labs are ordered ALPHABETICALLY; sorting them by best alpha was itself a
    ranking claim
  - the alpha cell loses its bold, which read as "the one to compare on"
  - the "Best alpha" card becomes "Alpha range", a fact rather than a podium

## Found while checking the boost mechanic: the return is not the balance

Chasing whether boosting touches scoring, I pulled portfolio values per round
and they did not reconcile with what we publish.

    agent                  published return    wallet over the same rounds
    gemini 3 pro vision          -15.72 %      264.87 -> 3.33     (-98.7 %)
    sonnet 4.5 vision            -25.22 %      229.40 -> 10.74    (-95.3 %)
    opus 4.5 chart               -19.22 %      297.64 -> 302.76   (+1.7 %)

The last line is the one that matters. We publish -19.22 % for an agent whose
wallet finished UP.

Our arithmetic is not wrong. Compounding each round's percentage is a
time-weighted return, which deliberately ignores money moved in or out between
rounds because the arena decides those and the agent does not. That is the
standard way to judge a manager whose cash flows it does not control, and it
is the right choice here.

But the page never said which of the two it was showing, and a reader assuming
the Return column was the money would have been wrong by twenty points. It
says so now, in the methodology, in a new FAQ entry and under the table.

## Capital is small and did not hold still

Median portfolio is a few hundred dollars an agent, which the page already
hinted at and now states as a column. The new finding is the instability: the
largest-to-smallest ratio across an agent's own rounds runs from 1.3 to about
190, and half the roster is above 5.

That is not cosmetic. Gas is a fixed cost per swap, so an agent trading the
same way on three dollars carries drag an agent on three hundred does not, and
for those agents the late rounds are not the same experiment as the early
ones. Published as trading_agent_capital_ratio.

I had written "two agents near 86" from an earlier sample that only looked at
final values; the full walk gives 1.3 to 190 with grok 4 chart the widest. The
copy carries the measured range.

## The boost mechanic, which started all this

No evidence it touches scoring: boosterAllocation is null, rewardsTge is
{agentPool: 0, userPool: 1e22} so the rewards go to the humans who boost
rather than to agents, and /competitions/{id}/boosts is 401. Not provable from
outside, but nothing points the other way and the capital walk shows balances
moving only through trading and withdrawals.

## Housekeeping

The spec is now at 17 metric panels of a maximum 17 and 9 ledger columns of a
maximum 9. Both at the ceiling. `own_t` was dropped to make room: data-only,
backed no column, and the checks read that metric straight from Prometheus
rather than through the panel. Anything further needs something removed.
@Flotapponnier
Flotapponnier merged commit 24a87ea into dev Oct 7, 2026
@Flotapponnier
Flotapponnier deleted the feat/284-tied-not-ranked branch November 3, 2026 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant