Skip to content

fix: preserve evaluation results when a verifier raises - #44

Open
Cookie0103 wants to merge 1 commit into
huggingface:mainfrom
Cookie0103:fix/isolate-evaluation-verifier-errors
Open

Cookie0103 wants to merge 1 commit into
huggingface:mainfrom
Cookie0103:fix/isolate-evaluation-verifier-errors

Conversation

@Cookie0103

Copy link
Copy Markdown

A verifier exception currently escapes after a successful agent execution and aborts evaluate_skill. For example, a file_contains check against a directory or a non-UTF-8 artifact prevents the remaining test cases and baseline phase from completing, and leaves that case without test_result.json.

Catch ordinary exceptions at the per-case verification boundary. Record a failed TestResult with the exception type/message while retaining the completed agent output, token/turn counts and full statistics. Persist through the existing result path so sibling cases and baseline evaluation can finish. Ordinary assertion failures retain validation_result; BaseException interruption/cancellation still propagates. README documents the distinction.

Validation:

  • Four new cases fail on unchanged main: unreadable UTF-8, directory input, command timeout and custom-validator error.
  • Integration regression covers both evaluation phases, healthy and mismatched sibling cases, aggregate rates, artifact reload and refinement feedback. A fifth case checks BaseException propagation.
  • Full suite: 90 passed. Required format, lint and typecheck scripts pass.
  • Deterministic backend fixtures replace model execution; actual filesystem, verifier and subprocess behavior is exercised. No live model/HF job was run.

Command descendant cleanup, subsequent verifiers within an errored case, and persistence-write recovery are unchanged.

AI assistance: implemented, tested and independently agent-reviewed with OpenAI Codex. The account owner authorized this contribution but has not personally reviewed or run these checks.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant