You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
P2: Add Semantic Drift research track using SZZ-backed historical evidence #47
semantic-drift, szz, leakmine, review, behavior-change, refactor-risk, research
Context
LeakFixMine already uses a careful anti-luck methodology: do not count "warning disappeared after fix" unless the fix causally touched the finding site. That same philosophy can support code review, but not by pretending SZZ can prove live behavior changes.
The correct role of SZZ here:
Offline: mine historical fixes and causally bind them to changed lines/sites.
Online: use that history as evidence/boost when a current PR has risky semantic drift.
Proposed direction
Create a Semantic Drift track alongside LeakFixMine.
Candidate categories:
null/guard removed before dereference;
exception path changed;
return value/constant changed;
transaction/lock/cleanup boundary removed;
dispose/using/finally removed;
state protocol violation introduced;
notification before restored invariant;
"refactor/no behavior change" claim contradicted by semantic delta.
Review formula
Sketch:
live_suspicion =
refactor_claimed
AND behavior_delta_present
AND not_explained_by_refactoring
AND optional(history_boost OR missing_tests OR public_api_touch)
SZZ contributes history_boost, not the live verdict.
Scope
Use deterministic patch/semantic signals first.
Use LLM only for explanation/summarization of deterministic evidence, not as a truth source.
Keep this in OwnAudit as mining/evidence/review research.
Own.NET should own any deterministic live semantic checks that become mature.
Acceptance criteria
Design doc describing driftmine/ or equivalent module boundary.
Candidate scoring model, similar in spirit to LeakFixMine but with behavior-drift categories.
At least one offline mined category, for example guard/null-fix or cleanup-boundary-fix.
Clear statement of what metrics do and do not mean.
Example PR-review report shape that cites:
claimed intent;
semantic delta;
historical evidence;
recommended reviewer focus.
Non-goals
Do not claim to prove behavior equivalence.
Do not gate CI from SZZ evidence alone.
Do not make an LLM reviewer the source of truth.
Why this matters
This can connect the existing LeakFixMine/SZZ work to practical code review: not "the bot vibes that refactor is risky", but "this PR matches a category of semantic drift that has historically produced fixes, and here is the deterministic evidence." Less horoscope, more forensics.
Interesting pipeline boundary: deterministic git-history statistics / SZZ candidate attribution and separate LLM root-cause classification + narrative; numerical metrics must not be overwritten by prose. The project explicitly makes metrics failures fail-hard, model narrative fail-soft, and publishes a metrics-only report if prose fails. Independent job budgets prevent historical scans from consuming the entire analysis window.
Fit with this issue: use as a model of separated evidence and interpretation, not as proof that SZZ establishes current semantic drift. Offline mined SZZ history can support a reviewer focus suggestion; an online 'behavior changed' verdict still needs a live deterministic witness. Candidate minimal experiment: one frozen historical fix cohort, timestamp/SHA-scoped SZZ evidence, uncertain/ambiguous origin status, side-by-side report with and without model explanation; compare audit accuracy, false attribution and operator usefulness. No CI gate based solely on SZZ, no new implementation authorized.
Activity
PhysShell commentedon Nov 21, 2026
2026-11-21 — KiroCrew fix-loop metrics as an SZZ reporting reference
Source: KiroCrew
.github/workflows/fix-loop-analysis.yml, plus research Page.Interesting pipeline boundary: deterministic git-history statistics / SZZ candidate attribution and separate LLM root-cause classification + narrative; numerical metrics must not be overwritten by prose. The project explicitly makes metrics failures fail-hard, model narrative fail-soft, and publishes a metrics-only report if prose fails. Independent job budgets prevent historical scans from consuming the entire analysis window.
Fit with this issue: use as a model of separated evidence and interpretation, not as proof that SZZ establishes current semantic drift. Offline mined SZZ history can support a reviewer focus suggestion; an online 'behavior changed' verdict still needs a live deterministic witness. Candidate minimal experiment: one frozen historical fix cohort, timestamp/SHA-scoped SZZ evidence, uncertain/ambiguous origin status, side-by-side report with and without model explanation; compare audit accuracy, false attribution and operator usefulness. No CI gate based solely on SZZ, no new implementation authorized.