SWE-bench Verified: a trial that changed nothing is graded as a patch, and resolves django__django-10097 #33

Priority

extreme

Components

• app/nullius (backend)
• app/web (routes)
• templates and static files
• docs and the site
• catalogue (benchmarks, agents, capabilities)
• the standard (ports, kits, relay, runs)
• CI, workflows and skills

Evidence

SWE-bench's harness grades a prediction, a patch. A prediction without one is never run: get_eval_report returns patch_is_None and the instance is not resolved. On the Nullius standard the agent's changes are tested where it made them, and the driver hands the grader a fixed string in the patch's place:

PREDICTION = "the agent's changes, made in the container"

So every trial reads patch_exists: true, whatever the agent did, and a trial that changed nothing is graded as if it had submitted a patch. For django__django-10097, whose graded tests pass without the fix, the null control resolves the instance:

django__django-10097: reward 1.0
resolved True  FAIL_TO_PASS {success: 438, failure: 0}  PASS_TO_PASS {success: 1427, failure: 0}

The benchmark's page files this under the upstream's limitations ("passes with no change"). On the upstream's harness an empty patch does not pass. The agent's patch is also kept nowhere in the trial: only the list of files that changed.

Screenshot

No response

Reproduction

1. nullius standard run swe-bench-verified nop on django__django-10097.
2. The trial's reward is 1.0 and its report says patch_exists: true.

Expected

The trial keeps the patch the agent made, and the grader is given it: a trial that changed nothing has no patch and resolves nothing, as on the upstream's harness.

Acceptance

• Before the eval script runs, the changes in /testbed since the instance's base commit (committed or not, new files included) are written as a patch and kept in the trial (verifier/model.patch).
• The grader's prediction carries that patch; with none, the report says patch_is_None and the instance is not resolved.
• The null control does not resolve django__django-10097; the reference control still does.
• The limitation about django__django-10097 says what is still true of it.

Dependencies and related

none