HumanEval's Original setup runs model-written code on the machine without asking #19

Priority

extreme

Components

• app/nullius (backend)
• app/web (routes)
• templates and static files
• docs and the site
• catalogue (benchmarks, agents, capabilities)
• the standard (ports, kits, relay, runs)
• CI, workflows and skills

Evidence

HumanEval's Original setup asks a model for the code and scores it with the upstream's check_correctness, which execs that code. A setup runs on local processes only (run.py refuses another backend for it) and, unlike an agent's local run, asks no --unsandboxed yes:

if backend == "local" and other.get("_kind") == "agent" and not unsandboxed:
    raise Refused(...)

The driver's check goes in a container only on the docker backend (NULLIUS_CHECK_IMAGE set by the run there). So nullius standard run humaneval original runs model-written code on the user's machine, as the user, with only the upstream's reliability_guard, which says of itself: "This function is NOT a security sandbox."

Screenshot

No response

Reproduction

1. On a machine with docker, nullius standard run humaneval original --model <any> --limit 2.
2. No yes is asked for.
3. The model's solution.py is executed by the driver's own process tree on the machine.

Expected

A setup's check that runs code runs it in a throwaway container with no network whenever docker answers, and on the machine only where there is no docker. The trial says which.

Acceptance

• A benchmark says its check runs the code it scores (score.runs_code), validated as true or false.
• On the local backend, a setup's run of such a benchmark gives its driver the base image as NULLIUS_CHECK_IMAGE when docker is usable, and nothing when it is not.
• An agent's local run and a benchmark whose check runs no code are unchanged.
• HumanEval's trial detail says where the check ran (check: container or this machine).
• HumanEval's limitation and docs/standard.md say the rule.

Dependencies and related

none