Repository navigation
fix(score): drop the snapshot-safety metric - #6
Merged
Merged
Conversation
meta git snapshot no longer exists, so the metric (25% of the grade) penalised every session with a destructive command and its suggestion told agents to run a command that fails. Remove it end to end, along with destructive_ops_detected, which only it used, and rescale the remaining weights 25/20/20/10 to 5:4:4:2 of 15 (about 33/27/27/13). Implements [[tasks/harmony-2768]]
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
meta git snapshotwas removed from meta (meta#115; #5 already removed the guard's snapshot hints).agent scorestill graded Snapshot safety at 25% of the overall grade: a destructive command only counted as safe ifmeta git snapshot createran in the 5 commands before it. That command no longer exists, so every session with a destructive command lost a quarter of its grade, and the suggestion told agents to run a command that fails.This drops the metric outright rather than replacing it: text matching can't reliably tell whether a commit or stash protected the work a later command destroys, and agent guard already blocks the worst destructive commands.
Removed end to end in
src/score.rs:snapshots_before_destructive,snapshot_safety_score/grade, the transcript counting, the grade, the report row, the suggestion and the matching JSON fields.destructive_ops_detectedwent too: only this metric and its "Destructive ops: N (M protected)" summary line used it. The per-commandBashCommand.is_destructiveflag in the JSON stays (it's per-command classification, not part of the metric). README metric list updated.Weights
The remaining weights keep their relative proportions (25:20:20:10 = 5:4:4:2):
Report before / after
Same 4-command transcript (
meta context,meta git status,git reset --hard HEAD,meta git commit -m x), old binary vs this branch:Verification
cargo fmt --check,cargo clippy --all-targets -- -D warnings,cargo test(126 passed) locally.test_overall_weights: a session with only bare git (meta ratio 0, everything else perfect) is D under the new weights (10/15 ≈ 0.67); under the old weights it was C (0.75), so the test pins the rescale. The oldtest_snapshot_protectionis gone; other tests just lose the snapshot argument.agent score --jsonoutput has nosnapshotkey.grep -rni snapshot .(excluding.git): no matches.Behavior change
Intended: overall grades change because the weights changed. Sessions with no destructive commands used to get a free 25% from this metric (
old = 0.25 + 0.75 × new), so they can grade lower now; sessions with destructive commands no longer lose that 25%.For a reviewer / human
Not changed here, but worth a look: the transcript structs use literal serde names
_uuid,_role,_id(and_text,_tool_use_id, ...). Claude Code transcripts useuuid/role/id, so as far as I can tell real transcripts never parse into tool calls andagent scorereports 0 tool calls. I needed those literal keys to build the demo transcript above. Out of scope for this PR.