Repository navigation
Description
Priority
extreme
Components
- app/nullius (backend)
- app/web (routes)
- templates and static files
- docs and the site
- catalogue (benchmarks, agents, capabilities)
- the standard (ports, kits, relay, runs)
- CI, workflows and skills
Evidence
Terminal-Bench's own harness has tmux in every task's container. On the Nullius standard a task runs in its own image as published, and most images have none (11 of the Dockerfiles install it). Two things fail for that reason alone.
The reference solution of git-multibranch (nullius standard run terminal-bench oracle, that task):
git-multibranch: error NonZeroAgentExitCodeError
/solution/solve.sh: line 54: tmux: command not found
Terminus 2, the benchmark's reference agent in the agents catalogue (only_for: terminal-bench), on regex-log with the docker backend:
regex-log: error NonZeroAgentExitCodeError
DEBUG:harbor.utils.logger:Installing: tmux
DEBUG:harbor.utils.logger:Installing tmux from source...
ERROR:harbor.utils.logger:Failed to install tmux from source
ERROR:terminus-2:terminus-2: RuntimeError: Failed to start
Terminus installs tmux itself when the image has none, and the agent's container has no way out but its model. So the benchmark's page says "Terminus-2, the authors' own agent, runs only on Harbor", while the catalogue lists it as fitting.
Screenshot
No response
Reproduction
nullius standard run terminal-bench oracle --dataset offline:git-multibranchends inNonZeroAgentExitCodeError.nullius standard run terminal-bench terminus-2 --model <any> --backend docker --dataset small: every task whose image has no tmux ends inNonZeroAgentExitCodeErrorbefore the first model call.
Expected
A task's container has tmux before anyone works in it, as on Terminal-Bench's own harness, and the agent still has no way out but its model.
Acceptance
- A unit's setup may carry a
preparestep, run as root in the unit's own container before the agent or a control starts. - A preparation's way out is the check's proxy, and its container is removed before the agent starts; a test with docker shows no route out and no
egressname afterwards. - A preparation that fails, or whose way out cannot be closed, is an
EnvironmentSetupError, not the agent's result. - Terminal-Bench's driver prepares each task with tmux when its image has none, and leaves an image that has it alone.
- The reference solution of
git-multibranchpasses. - Terminus 2 runs a task whose image has no tmux.
- The benchmark's limitations and
upstream-settings.jsonno longer say Terminus 2 runs only on Harbor, nor that tmux is missing.
Dependencies and related
The offline dataset's recorded oracle proof (45/53) predates this and still counts git-multibranch as failing; it is not re-run here.
Activity