Repository navigation
Description
Classification
- Type: bounded research / evaluation-infrastructure spike
- Priority: P2 / backlog. Record the idea now; do not start implementation by default.
- Area: S6 candidate generation + reranking, offline experiment reporting
- Inspiration: Trusted-AI/AIF360 subgroup/intersectional analysis as a methodology only, not demographic fairness and not a library integration
- Related: Rerank provenance: preserve every candidate and record explicit rejection reasons #104 (full candidate evaluation provenance, prerequisite for trustworthy denominators), Benchmark-first symbolic retrieval: hard constraints, ablations and curated relevance fixtures #107 (S7 retrieval benchmark; separate problem), S9 Counterfactual Preference Lab: pairwise A/B feedback over single-axis candidate deltas #106 (S9 counterfactual preferences; later exposure-bias consumer)
- Architecture: ADR-0034 (accepted experiment identities and comparable-only metrics), ADR-0032 (offline Reachability Lab owns corpus holdout)
Why
An aggregate generation/rerank score may hide concentrated losses in some musically meaningful, eligible candidate groups. For example, complex syncopated ideas could be produced but consistently absent from Top-K. That observation alone does not establish a ranking defect: their scores, constraints or musical suitability may legitimately differ.
We need a small, reproducible candidate-funnel audit that separates:
generated -> evaluable / rejected (typed reason) -> ranked -> Top-K
|
later S9:
exposed -> auditioned
-> preferred
In particular, absence from Top-K is not equivalent to being generated rarely; rejection at measurement time is not a musical dislike; absence of S9 feedback is not a negative preference.
Scope: one offline S6 pilot
- Prerequisite/readiness check: inspect Rerank provenance: preserve every candidate and record explicit rejection reasons #104 and the canonical path that feeds the runner. Do not silently use
rerank_candidates()ranked-only output as the full generated population. If candidate accounting is still lossy, stop with BLOCKED-BY-Rerank provenance: preserve every candidate and record explicit rejection reasons #104 (or explicitly limit the experiment to a non-scored inventory that does not pretend to measure the full funnel). Avoid reimplementing Rerank provenance: preserve every candidate and record explicit rejection reasons #104 here. - Reuse
griff-experimentand the accepted variant × information-regime model in ADR-0034. Use fixed generation requests, seeds, corpus snapshot identities and one explicit evaluation context. No parallel experiment framework, production code integration or changes to S6 scoring. - Record one outcome per generated candidate, with stable provenance: experiment/cell/recipe identity, strategy, seed, applicable input/control and observed musical features, score-axis values, typed measurement rejection (if any), and rank/Top-K membership. Preserve absent, unsupported and unmeasurable facts as unknown; never infer fretting-dependent techniques from score notes that lack string/fret realization.
- Begin with a small, explicitly declared taxonomy from facts the current generator can actually observe: strategy, gesture enabled/disabled, rhythmic density/syncopation only if measured consistently, register, phrase length, etc. Add technique tags or intersections only when their source and meaning are demonstrably available. Do not invent a generic notion of musical groups.
- Show for each supported tag and a few predeclared, sufficiently populated intersections:
- generated / evaluated / rejected-with-reason / ranked / Top-K raw counts and denominators;
Top-K / rankedandTop-K / eligiblerates where applicable, relative to the same request's actual candidate pool;- skipped/missing measurements and coverage separately;
- cross-seed and (when corpus-backed) source-song-aware summaries to avoid pretending correlated variants/chunks are independent.
- Manually inspect one or two reproducible large count-backed imbalances: policy explanations, hard constraints, musical examples/A-B, and possible artifact/denominator errors. An independent curator judgment or task-specific relevance oracle is required before claiming a group contains unfairly suppressed good material. Report an unexplained distribution shift as a hypothesis, not a confirmed musical quality failure.
Methodological safeguards
- Not representational parity: swancore-focused generation has no obligation to output equal shares of tapping, syncopation, palm-muted patterns, etc. A rare class in Top-K is not, by itself, a defect.
- No changing ruler: ADR-0034 metric comparability is mandatory. A cell's own novelty/objective score under a different information regime cannot be interpreted as a comparable quality evaluation.
- No sample leakage: holdout/population preparation stays in the Reachability Lab per ADR-0032; group same-song material when relevant, and use pinned corpus/source identities.
- No denominator tricks: measurement failure must not disappear from the generated population; identical/near-duplicate material should not inflate perceived diversity; small/empty slices show counts and insufficient-evidence status, not confidently ranked percentages.
- No post-hoc subgroup fishing presented as discovery: declare a limited initial taxonomy/selection rule before reading results; label additional explorations explicitly.
- No feedback claims yet: generated/ranked/Top-K is observable in S6. Exposed/auditioned/preferred is a future S9 extension, not something to reconstruct from sparse session-local favorites.
- No new quality scalar, score equalization, ML/RL training, automatic rerank reweighting, or AIF360 dependency.
Outputs / acceptance
- Document whether Rerank provenance: preserve every candidate and record explicit rejection reasons #104 provides lossless per-candidate outcomes; if not, record a typed BLOCKED decision and do not fake the missing population.
- Reproducible, bounded headless fixture run over fixed requests/seeds, using existing experiment infrastructure and pinned spec/corpus/policy/evaluation identities.
- Machine-readable table (JSON/CSV) and concise human report with stage-by-stage counts, reason breakdowns, missingness, and only supported subgroup metrics.
- At least one synthetic accounting fixture proving
N generated = ranked + rejectedfor a full-evaluation path and no silent loss; plus deterministic replay and a sparse/unknown-subgroup case. - One or two manually audited imbalances with candidate examples, provenance and honest classification: intended policy behavior, weak evidence, measurement bias, known issue, or independently corroborated potential musical miss.
- Short GO / STOP / BLOCKED conclusion, including what (if anything) should be added as targeted regression tests or a follow-on S9 Counterfactual Preference Lab: pairwise A/B feedback over single-axis candidate deltas #106 S9 exposure-bias experiment.
Stop / promotion gate
Treat this as a timeboxed research spike once the project is active again, not an immediate implementation assignment.
- GO: independent evidence identifies a reproducible, actionable weak slice or the small auditor yields clear incremental regression value.
- STOP: results are only aesthetically interesting category proportions, the corpus is too small, groups are unobservable, or the finding is already covered elsewhere.
- BLOCKED: cannot account for the generated population (notably Rerank provenance: preserve every candidate and record explicit rejection reasons #104), validly group examples, or compare the measurements.
Do not gate CI on these metrics or change production ranking until an independently validated improvement exists. #107 remains retrieval evaluation, not the owner of this S6 audit; #106 can later reuse its funnel vocabulary for exposure/selection-bias controls.
Activity