Skip to content

fix(score): drop the snapshot-safety metric - #6

Merged
gold-silver-copper merged 1 commit into
mainfrom
fix/score-drop-snapshot-metric
Oct 26, 2026
Merged

gold-silver-copper merged 1 commit into
mainfrom
fix/score-drop-snapshot-metric

Conversation

@gold-silver-copper

Copy link
Copy Markdown
Contributor

What and why

meta git snapshot was removed from meta (meta#115; #5 already removed the guard's snapshot hints). agent score still graded Snapshot safety at 25% of the overall grade: a destructive command only counted as safe if meta git snapshot create ran in the 5 commands before it. That command no longer exists, so every session with a destructive command lost a quarter of its grade, and the suggestion told agents to run a command that fails.

This drops the metric outright rather than replacing it: text matching can't reliably tell whether a commit or stash protected the work a later command destroys, and agent guard already blocks the worst destructive commands.

Removed end to end in src/score.rs: snapshots_before_destructive, snapshot_safety_score/grade, the transcript counting, the grade, the report row, the suggestion and the matching JSON fields. destructive_ops_detected went too: only this metric and its "Destructive ops: N (M protected)" summary line used it. The per-command BashCommand.is_destructive flag in the JSON stays (it's per-command classification, not part of the metric). README metric list updated.

Weights

The remaining weights keep their relative proportions (25:20:20:10 = 5:4:4:2):

Metric Before After
Meta-command ratio 25% 33% (5/15)
Workspace discovery 20% 27% (4/15)
Snapshot safety 25% —
Cross-repo awareness 20% 27% (4/15)
Guard effectiveness 10% 13% (2/15)
// before
let overall = (meta_ratio * 0.25) + (discovery_score * 0.20) + (snapshot_score * 0.25)
    + (awareness_score * 0.20) + (guard_score * 0.10);
// after
let overall =
    (meta_ratio * 5.0 + discovery_score * 4.0 + awareness_score * 4.0 + guard_score * 2.0) / 15.0;

Report before / after

Same 4-command transcript (meta context, meta git status, git reset --hard HEAD, meta git commit -m x), old binary vs this branch:

before: # Session Score: demo (D)            after: # Session Score: demo (B)
| Meta-command ratio | 67% | D | 25% |        | Meta-command ratio | 67% | D | 33% |
| Workspace discovery | 100% | A ✓ | 20% |    | Workspace discovery | 100% | A ✓ | 27% |
| Snapshot safety | 0% | F ✗ | 25% |
| Cross-repo awareness | 100% | A ✓ | 20% |   | Cross-repo awareness | 100% | A ✓ | 27% |
| Guard effectiveness | 100% | A ✓ | 10% |    | Guard effectiveness | 100% | A ✓ | 13% |
- Destructive ops: 1 (0 protected)
- Only 0/1 destructive operations were preceded by snapshots (1 unprotected).
  Use `meta git snapshot create` before batch changes.

Verification

  • cargo fmt --check, cargo clippy --all-targets -- -D warnings, cargo test (126 passed) locally.
  • New test_overall_weights: a session with only bare git (meta ratio 0, everything else perfect) is D under the new weights (10/15 ≈ 0.67); under the old weights it was C (0.75), so the test pins the rescale. The old test_snapshot_protection is gone; other tests just lose the snapshot argument.
  • Ran old and new binaries on the transcript above; agent score --json output has no snapshot key.
  • grep -rni snapshot . (excluding .git): no matches.

Behavior change

Intended: overall grades change because the weights changed. Sessions with no destructive commands used to get a free 25% from this metric (old = 0.25 + 0.75 × new), so they can grade lower now; sessions with destructive commands no longer lose that 25%.

For a reviewer / human

Not changed here, but worth a look: the transcript structs use literal serde names _uuid, _role, _id (and _text, _tool_use_id, ...). Claude Code transcripts use uuid/role/id, so as far as I can tell real transcripts never parse into tool calls and agent score reports 0 tool calls. I needed those literal keys to build the demo transcript above. Out of scope for this PR.

meta git snapshot no longer exists, so the metric (25% of the grade)
penalised every session with a destructive command and its suggestion
told agents to run a command that fails. Remove it end to end, along
with destructive_ops_detected, which only it used, and rescale the
remaining weights 25/20/20/10 to 5:4:4:2 of 15 (about 33/27/27/13).

Implements [[tasks/harmony-2768]]
@coderabbitai

coderabbitai Bot commented Oct 26, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: fac3f274-c7c3-425e-8ba4-de40e32b2842
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@gold-silver-copper
gold-silver-copper merged commit 38a81a4 into main Oct 26, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant