Repository navigation
feat(tournament): complete phase 4a rehearsal tournament, scoring, and arbiter - #27
Conversation
…tion Add ProbeSampler and detect_recovery to observe candidate twin environments under mirrored traffic and detect SLO recovery. Samples at 1 Hz using ObservabilityAdapter, enforces a 20-second cold-start warm-up exclusion window, detects recovery at 15 consecutive healthy samples, and times out at 180 seconds. Implements build-plan step A4.1.
…selines Add BlastCoordinator, EnvironmentBaseline, and pure blast radius evaluation functions to tournament/blast.py. Captures per-environment pre-apply telemetry baselines before candidate plans are applied, evaluates >10% service-level degradation across latency and error rate, and calculates transitive blast radius and downstream error delta. Implements build-plan step A4.2.
… rules Add DeterministicCandidateScorer and pure candidate scoring functions in tournament/scorer.py. Calculates normalized subscores for recovery time, blast radius, downstream error delta, and runtime invariant violations, combining them via weights from config/scoring.yaml and global settings. Enforces hard disqualification rules for incomplete evidence (drop ratio > 0.05, probe samples < 60), unready twins, and hard runtime invariant violations (K6, K10). Adds property-based tests verifying recovery time monotonicity, non-viability of disqualified candidates, and NO_ACTION dominance when remediations exhibit negative side effects. Implements build-plan step A4.3.
Add AdvisoryLLMJudge and FakeLLMJudge in tournament/judge.py and tournament/fakes.py to support the parallel advisory LLM scoring path specified in ADR-017 and architecture §2.8. Serializes candidate evidence and optional remediation plans without leaking deterministic scores, weights, composites, or disqualification status to the LLM. Queries OpenRouter with structured JSON output, sanitizing the resulting rankings to eliminate hallucinations, deduplicate entries, and ensure total candidate ordering. Implements top-1 judge agreement computation for evaluation Metric 5. Defines LLMJudge Protocol in tournament/api.py and provides comprehensive unit tests achieving 100% line and branch coverage across the tournament package. Implements build-plan step A4.4.
Enforce deterministic arbitration across candidate rehearsal evidence with the 0.15 ambiguity margin per ADR-018 and architecture §2.7-§2.8. Guarantee that the winner derives strictly from deterministic scores, with an advisory LLM judge evaluation recorded for agreement metrics without influencing outcomes. Add ust tournament replay CLI subcommand and test fixtures for three-candidate decided outcomes, near-tie ambiguity, and high-drop candidate disqualification. Implements build-plan step A4.5.
Two-Axis Code Review Findings (
|
AGENTS.md §6.6 strictly requires hypothesis for property tests asserting scorer monotonicity and disqualification invariants. Adds hypothesis to dev dependencies and converts property tests to use hypothesis strategies. Refines build-plan step A4.3 and Checkpoint A4.
Resolves all 17 review findings (8 Standards, 9 Spec) across the Phase 4A tournament implementation: - Separates applied_at from forked_at and removes speculative recovery_time_mode in probe sampler. - Eliminates unrequested downstream aggregation modes and bare exception handling in blast calculator. - Moves ambiguity margin and service request shares into scoring.yaml, eliminating fallback defaults in safety path. - Removes unrequested plans parameter from advisory LLM judge. - Reorders baseline capture before probing and passes mirror registry stats in RehearsalTournament. - Enforces pure mathematical winner derivation from scores and evaluates ambiguity margin globally. - Enforces sibling import boundaries and typed exceptions in replay CLI. - Removes all unused type imports and maintains 100% line and branch test coverage. Implements build-plan step A4.
Code Review Findings Resolved ✅All 17 findings (8 Standards, 9 Spec) identified during the code review have been addressed in commits 1. Standards Axis Resolutions
2. Spec Axis Resolutions
Verification
|
Build-Plan Step Implemented
tournament/probe.py)tournament/blast.py)config/scoring.yaml, disqualification rules (tournament/scorer.py)tournament/judge.py)scores, andust tournament replayCLI (tournament/arbiter.py,cli.py)Checkpoint Execution & Output
All Checkpoint A4 commands were executed live and verified:
Full suite validation:
ADRs Relied On
AMBIGUOUS).Mock Registry Disclosures
# MOCKED:markers added; mock parity check clean).Documentation Updates
docs/01-decisions.md§ADR-017, ADR-018,docs/02-architecture.md§2.6-§2.8, anddocs/04-build-plan.md§Phase 4A;CHECKLIST.mdupdated with completed step A4.5 and Checkpoint A4 assertions).Definition of Done
make checkgreen (ruff, mypy --strict, import-linter, pytest)# MOCKED:marker without a row in docs/MOCKS.md