Let a benchmark say which tasks need the internet, so Terminal-Bench's 36 left-out tasks can run #26

Priority

extreme

Components

• app/nullius (backend)
• app/web (routes)
• templates and static files
• docs and the site
• catalogue (benchmarks, agents, capabilities)
• the standard (ports, kits, relay, runs)
• CI, workflows and skills

Today

On the Nullius standard an agent's only way out is its model: its container is on an internal network and the relay is its one door. A task whose solution needs the internet therefore cannot be solved, and Terminal-Bench 2.0 leaves such tasks out: of the 89 tasks at the pinned commit the driver runs 53 (OFFLINE), and the other 36 are not offered. 34 of those 36 reference solutions run a download command (apt-get, pip install, curl, git clone, ...); the other two are sparql-university and winning-avg-corewars. The benchmark's limitation says it: "The 36 tasks whose solution downloads something are left out, so 53 of the 89 run, and the agent cannot install tools." SWE-bench Verified says the same of its agents.

The pieces a way out needs are already there for the check: a proxy container that connects to public addresses only (standard/egress.py, Box.online), and since the preparation step, a way to take it away again (Box.offline).

Wanted

The benchmark says, per task, whether the task needs the internet; a run cannot ask for it. Decided by the owner: it is up to the benchmark to see whether it needs it.

• A unit's setup may say internet: true. Its agent is then given the check's proxy for the length of its work, named to its process (HTTP_PROXY, HTTPS_PROXY, with the relay and the benchmark's servers reached directly), and the proxy is removed before the check starts. What it let through and refused is kept in the trial (agent/egress.log).
• Every other unit's agent reaches nothing but its model, as today.
• Terminal-Bench marks its 36 such tasks, gives their reference solutions the same way out, and offers all 89 as a dataset. SWE-bench Verified marks none: an instance's fix is public.

Acceptance

• A unit's setup may carry internet, validated as true or false, and only for a unit that runs in its own image.
• With it, the agent's process is given the proxy, the relay stays reachable directly, and the trial keeps what the proxy let through and refused.
• Without it, nothing changes: the agent has no way out but its model.
• The proxy is gone before the check starts, and a proxy already up is not started twice.
• Terminal-Bench's driver marks its 36 tasks, pins their images by digest, and offers a dataset of all 89.
• Terminal-Bench's datasets are proven by their controls, with what still fails written in the upstream limitations.
• The benchmark's page and docs/standard.md say the rule.

Dependencies and related

Builds on the preparation step (Box.offline) added for Terminal-Bench's tmux.