Skip to content

Three limitation lines that are an option, a dangling reference and a test fixture #22

Description

@aitianshi

Priority

medium

Components

  • app/nullius (backend)
  • app/web (routes)
  • templates and static files
  • docs and the site
  • catalogue (benchmarks, agents, capabilities)
  • the standard (ports, kits, relay, runs)
  • CI, workflows and skills

Today

Three lines of the bundled benchmarks do not do what the page's rule asks of a limitation (it must affect a choice, a required action or the reading of a result):

  • Terminal-Bench lists "Each task's prebuilt image is pulled by default. The image option builds it here instead..." as a limitation. It describes an option.
  • Terminal-Bench's upstream-settings.json says of environment.storage_mb that it is "said in the limitations". No limitation says it.
  • τ³-bench says "The mock domain and voice mode are not run." The mock domain is the upstream's test fixture (10 tasks for its own tests), not part of the benchmark or its leaderboard.

Wanted

The image line is under Good to know; the settings inventory no longer points at a limitation that does not exist; τ³-bench's line names voice mode only.

Acceptance

  • Terminal-Bench's image line is in good_to_know, not limitations.
  • The storage_mb row of Terminal-Bench's settings inventory does not refer to the limitations.
  • τ³-bench's limitation does not mention the mock domain.
  • The Supported pages are regenerated.

Dependencies and related

none

Activity

  1. self-assigned this
    on Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions