Repository navigation
Release 0.34.0 - #2245
Merged
Merged
Release 0.34.0#2245
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
0.34.0 adds Custom runner Arms for process Experiments, with preflight checks, sandboxed execution and retained Session accounting (#2203).
thirdshift evals check-runner <name-or-file.toml> [--suite <suite> --task <task>] [--json]checks runner inputs, credentials and Model pricing, with an optional sandboxed Trial that creates no Experiment record. Available throughPOST /api/v1/evals/check-runnertoo (Add evals check-runner, its Console API twin, and Custom runner documentation #2239).--baseline-runnerand--candidate-runnerfor explicit process suites, freezing manifests and mapped files with provenance. Setup format 13 makes commit/Harness fields nullable; older builds cannot read Custom-runner Experiments (Freeze Custom runner Arms and run process Trials #2194, Spec: Custom-runner Arms for process Experiments #2203).[providers]keys and provider hosts. Panel voting withholds same-family votes. Harden host file handling and bound recorder resources (Enclose Custom runner Trials in declared multi-provider sandboxes #2205, Spec: Custom-runner Arms for process Experiments #2203, Record Custom runner sessions and enforce Trial accounting #2227).record_commandorsessions.jsonfor Trial Peek, Waterfall and export. Quota usage at 85% pauses Experiments; web-search evidence voids native and Custom Trials. Missing Session accounting voids valid runner-reported outcomes, while crashes and timeouts remain failed (Record Custom runner sessions and enforce Trial accounting #2227).Site and docs: document Custom runners and preflight checks, update site pages and the generated manual, and deduplicate release Pass-skip guidance (#2239, #2244).
Version diff