Skip to content

Research: stratified candidate-funnel audit for Griff S6 (bounded pilot) #215

Open

Description

@PhysShell

Classification

Why

An aggregate generation/rerank score may hide concentrated losses in some musically meaningful, eligible candidate groups. For example, complex syncopated ideas could be produced but consistently absent from Top-K. That observation alone does not establish a ranking defect: their scores, constraints or musical suitability may legitimately differ.

We need a small, reproducible candidate-funnel audit that separates:

generated -> evaluable / rejected (typed reason) -> ranked -> Top-K
                                                        |
                                                    later S9:
                                               exposed -> auditioned
                                                        -> preferred

In particular, absence from Top-K is not equivalent to being generated rarely; rejection at measurement time is not a musical dislike; absence of S9 feedback is not a negative preference.

Scope: one offline S6 pilot

  1. Prerequisite/readiness check: inspect Rerank provenance: preserve every candidate and record explicit rejection reasons #104 and the canonical path that feeds the runner. Do not silently use rerank_candidates() ranked-only output as the full generated population. If candidate accounting is still lossy, stop with BLOCKED-BY-Rerank provenance: preserve every candidate and record explicit rejection reasons #104 (or explicitly limit the experiment to a non-scored inventory that does not pretend to measure the full funnel). Avoid reimplementing Rerank provenance: preserve every candidate and record explicit rejection reasons #104 here.
  2. Reuse griff-experiment and the accepted variant × information-regime model in ADR-0034. Use fixed generation requests, seeds, corpus snapshot identities and one explicit evaluation context. No parallel experiment framework, production code integration or changes to S6 scoring.
  3. Record one outcome per generated candidate, with stable provenance: experiment/cell/recipe identity, strategy, seed, applicable input/control and observed musical features, score-axis values, typed measurement rejection (if any), and rank/Top-K membership. Preserve absent, unsupported and unmeasurable facts as unknown; never infer fretting-dependent techniques from score notes that lack string/fret realization.
  4. Begin with a small, explicitly declared taxonomy from facts the current generator can actually observe: strategy, gesture enabled/disabled, rhythmic density/syncopation only if measured consistently, register, phrase length, etc. Add technique tags or intersections only when their source and meaning are demonstrably available. Do not invent a generic notion of musical groups.
  5. Show for each supported tag and a few predeclared, sufficiently populated intersections:
    • generated / evaluated / rejected-with-reason / ranked / Top-K raw counts and denominators;
    • Top-K / ranked and Top-K / eligible rates where applicable, relative to the same request's actual candidate pool;
    • skipped/missing measurements and coverage separately;
    • cross-seed and (when corpus-backed) source-song-aware summaries to avoid pretending correlated variants/chunks are independent.
  6. Manually inspect one or two reproducible large count-backed imbalances: policy explanations, hard constraints, musical examples/A-B, and possible artifact/denominator errors. An independent curator judgment or task-specific relevance oracle is required before claiming a group contains unfairly suppressed good material. Report an unexplained distribution shift as a hypothesis, not a confirmed musical quality failure.

Methodological safeguards

  • Not representational parity: swancore-focused generation has no obligation to output equal shares of tapping, syncopation, palm-muted patterns, etc. A rare class in Top-K is not, by itself, a defect.
  • No changing ruler: ADR-0034 metric comparability is mandatory. A cell's own novelty/objective score under a different information regime cannot be interpreted as a comparable quality evaluation.
  • No sample leakage: holdout/population preparation stays in the Reachability Lab per ADR-0032; group same-song material when relevant, and use pinned corpus/source identities.
  • No denominator tricks: measurement failure must not disappear from the generated population; identical/near-duplicate material should not inflate perceived diversity; small/empty slices show counts and insufficient-evidence status, not confidently ranked percentages.
  • No post-hoc subgroup fishing presented as discovery: declare a limited initial taxonomy/selection rule before reading results; label additional explorations explicitly.
  • No feedback claims yet: generated/ranked/Top-K is observable in S6. Exposed/auditioned/preferred is a future S9 extension, not something to reconstruct from sparse session-local favorites.
  • No new quality scalar, score equalization, ML/RL training, automatic rerank reweighting, or AIF360 dependency.

Outputs / acceptance

  • Document whether Rerank provenance: preserve every candidate and record explicit rejection reasons #104 provides lossless per-candidate outcomes; if not, record a typed BLOCKED decision and do not fake the missing population.
  • Reproducible, bounded headless fixture run over fixed requests/seeds, using existing experiment infrastructure and pinned spec/corpus/policy/evaluation identities.
  • Machine-readable table (JSON/CSV) and concise human report with stage-by-stage counts, reason breakdowns, missingness, and only supported subgroup metrics.
  • At least one synthetic accounting fixture proving N generated = ranked + rejected for a full-evaluation path and no silent loss; plus deterministic replay and a sparse/unknown-subgroup case.
  • One or two manually audited imbalances with candidate examples, provenance and honest classification: intended policy behavior, weak evidence, measurement bias, known issue, or independently corroborated potential musical miss.
  • Short GO / STOP / BLOCKED conclusion, including what (if anything) should be added as targeted regression tests or a follow-on S9 Counterfactual Preference Lab: pairwise A/B feedback over single-axis candidate deltas #106 S9 exposure-bias experiment.

Stop / promotion gate

Treat this as a timeboxed research spike once the project is active again, not an immediate implementation assignment.

  • GO: independent evidence identifies a reproducible, actionable weak slice or the small auditor yields clear incremental regression value.
  • STOP: results are only aesthetically interesting category proportions, the corpus is too small, groups are unobservable, or the finding is already covered elsewhere.
  • BLOCKED: cannot account for the generated population (notably Rerank provenance: preserve every candidate and record explicit rejection reasons #104), validly group examples, or compare the measurements.

Do not gate CI on these metrics or change production ranking until an independently validated improvement exists. #107 remains retrieval evaluation, not the owner of this S6 audit; #106 can later reuse its funnel vocabulary for exposure/selection-bias controls.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions