Skip to content

Terminal-Bench tasks have no tmux: git-multibranch's reference solution and Terminus 2 both fail #21

Open

Description

@aitianshi

Priority

extreme

Components

  • app/nullius (backend)
  • app/web (routes)
  • templates and static files
  • docs and the site
  • catalogue (benchmarks, agents, capabilities)
  • the standard (ports, kits, relay, runs)
  • CI, workflows and skills

Evidence

Terminal-Bench's own harness has tmux in every task's container. On the Nullius standard a task runs in its own image as published, and most images have none (11 of the Dockerfiles install it). Two things fail for that reason alone.

The reference solution of git-multibranch (nullius standard run terminal-bench oracle, that task):

git-multibranch: error NonZeroAgentExitCodeError
/solution/solve.sh: line 54: tmux: command not found

Terminus 2, the benchmark's reference agent in the agents catalogue (only_for: terminal-bench), on regex-log with the docker backend:

regex-log: error NonZeroAgentExitCodeError
DEBUG:harbor.utils.logger:Installing: tmux
DEBUG:harbor.utils.logger:Installing tmux from source...
ERROR:harbor.utils.logger:Failed to install tmux from source
ERROR:terminus-2:terminus-2: RuntimeError: Failed to start

Terminus installs tmux itself when the image has none, and the agent's container has no way out but its model. So the benchmark's page says "Terminus-2, the authors' own agent, runs only on Harbor", while the catalogue lists it as fitting.

Screenshot

No response

Reproduction

  1. nullius standard run terminal-bench oracle --dataset offline: git-multibranch ends in NonZeroAgentExitCodeError.
  2. nullius standard run terminal-bench terminus-2 --model <any> --backend docker --dataset small: every task whose image has no tmux ends in NonZeroAgentExitCodeError before the first model call.

Expected

A task's container has tmux before anyone works in it, as on Terminal-Bench's own harness, and the agent still has no way out but its model.

Acceptance

  • A unit's setup may carry a prepare step, run as root in the unit's own container before the agent or a control starts.
  • A preparation's way out is the check's proxy, and its container is removed before the agent starts; a test with docker shows no route out and no egress name afterwards.
  • A preparation that fails, or whose way out cannot be closed, is an EnvironmentSetupError, not the agent's result.
  • Terminal-Bench's driver prepares each task with tmux when its image has none, and leaves an image that has it alone.
  • The reference solution of git-multibranch passes.
  • Terminus 2 runs a task whose image has no tmux.
  • The benchmark's limitations and upstream-settings.json no longer say Terminus 2 runs only on Harbor, nor that tmux is missing.

Dependencies and related

The offline dataset's recorded oracle proof (45/53) predates this and still counts git-multibranch as failing; it is not re-run here.

Activity

  1. self-assigned this
    on Nov 10, 2026
  2. added theissue type on Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions