Skip to content

Research: benchmark CLMR audio embeddings against Griff symbolic riff similarity #216

Open

Description

@PhysShell

Classification

Motivation and hypothesis

Griff is a symbolic, swancore-first guitar-riff engine. Its canonical Guitar Pro / score model is the source of truth, and MIDI is an interchange boundary; Griff is not an audio synthesizer.

It already has explainable symbolic baselines:

  • core/src/similarity.rs: 15 named axes (similarity v3) for corpus-chunk retrieval;
  • core/src/novelty.rs: symbolic quotation / n-gram novelty;
  • core/src/rerank.rs: four closure axes plus two novelty axes for candidate ranking.

CLMR learns representations from music waveforms with contrastive learning (SampleCNN/SimCLR). A possible, unproven benefit is finding perceptual neighbours or near-duplicates that symbolic axes miss. The inverse risk is that cosine similarity measures timbre, mix, velocity or rendering artefacts rather than riff structure.

Research question: on short swancore-style guitar phrases, do CLMR audio embeddings add enough musically relevant signal beyond Griff's current symbolic features to justify maintaining an audio-rendering/evaluation pipeline?

The original CLMR paper evaluates music classification, not symbolic riff retrieval, riff quality, playability, or preference prediction. Treat transfer to Griff as a hypothesis.

Scope and sequencing

  1. Read/reuse Benchmark-first symbolic retrieval: hard constraints, ablations and curated relevance fixtures #107's versioned retrieval fixtures, intents, relevance grades and evaluation contract where applicable. If Benchmark-first symbolic retrieval: hard constraints, ablations and curated relevance fixtures #107 is not implemented, this issue may only prepare a standalone, explicitly non-comparable research harness; do not present a new metric as a validated retrieval improvement.
  2. Pin a small, legally reusable or synthetic set (suggested pilot: 30–50 source riffs, plus declared controlled variants), with stable IDs, corpus/fixture digests and source-song group IDs.
  3. Render the same symbolic material to audio with a pinned instrument, sample library, synth, sample rate, tempo and effects chain. Store render/config hashes and tool versions; do not commit unauthorized audio assets.
  4. Start with a frozen pre-trained CLMR checkpoint. Extract encoder embeddings (not classification probabilities); fix windowing, hop, normalization, aggregation and distance metric as versioned experimental parameters.
  5. Compare against Griff's current similarity v3 baseline (and appropriate novelty/rerank baseline for the particular task), simple audio-feature sanity baselines if useful, CLMR alone, and a late fusion variant. Optional follow-on: modern audio representation baseline such as MERT, only if warranted by a concrete failure or improvement.
  6. Report results and recommendation. No change to production generation, reranking weights, corpus schema, UI or plugin in this issue.

Adversarial fixture design

Use paired controls so the network cannot win by detecting a sample library:

  • Same notes/rhythm, different timbre: change guitar patch, gain, distortion/EQ, reverb and velocity independently.
  • Different riff, same timbre: different rhythm, contour or note ordering with identical rendering.
  • Near variants: transposition, octave shift, slight rhythmic displacement, ornamentation, motif substitution.
  • Same rhythm / different pitches and same contour / different rhythm.
  • Short versus multi-bar phrases: verify whether window aggregation changes rankings.
  • Same-song / same-source leakage: explicitly partition related variants and report when a result is trivially copied from the query.
  • Hard constraint failures: a perceptually close recording must never override tuning, range, technique, meter, or playability restrictions.

Declare each query's retrieval intent (e.g. near-variant, rhythmic analogue, technical analogue, contrast candidate). Similarity is not one universal musical property. Keep human/curator judgements independent of the embeddings.

Measurements

At minimum report, per intent and overall:

  • Recall@K and nDCG@K over graded relevance (plus MRR where appropriate);
  • hard-constraint violation rate after constraint filtering (must remain zero);
  • within-source leakage / near-duplicate rate and diversity@K;
  • per-query improvement/regression and uncertainty, not just one mean;
  • cosine-distance behaviour on same-riff/different-timbre vs different-riff/same-timbre pairs;
  • sensitivity to render preset, short windows, overlap, pooling, pitch transformations and embedding normalization;
  • inference time, memory and external deployment/dependency cost.

For a rerank claim, evaluate held-out human preference/utility labels and compare to the existing six-axis generation_rerank policy; retrieval relevance alone does not validate reranking generated candidates.

Avoid using audio-only cosine distance to infer articulation correctness, playability, harmonic validity or intrinsic musical quality.

Reproducibility / implementation concerns

CLMR's checked upstream config currently uses 22,050 Hz and 59,049 samples (~2.68 s per input), so multi-bar riffs require an explicit crop/aggregation policy. Upstream requirements.txt pins torch==1.9.0; isolate that environment instead of imposing it on the Rust workspace. The repository offers pre-trained model checkpoints and an export.py for ONNX, but its exporter also expects a finetuner checkpoint even when exporting the encoder-only artifact; verify export correctness, shapes and parity against the PyTorch encoder before treating ONNX as a usable integration path.

  • Python/PyTorch/ONNX may be offline tooling only for this spike.
  • Do not add a heavyweight ML runtime to the Griff Rust workspace, WASM cockpit or real-time CLAP path.
  • Preserve deterministic Griff generation under fixed seed and unchanged canonical score semantics.
  • Pin checkpoint hash, upstream commit, dependency lock, preprocessing version, fixture digest and render-chain configuration.
  • If new code is introduced, follow AGENTS.md TDD rules and keep all repo documentation in English.

Deliverables / acceptance

  • Small legally safe, intent-labelled adversarial fixture set with stable IDs and source-leakage policy.
  • Reproducible offline CLMR encoder extraction script with exact checkpoint/config/preprocessing manifest.
  • Validated fixed-window and multi-window aggregation behaviour.
  • Symbolic baseline vs CLMR vs late-fusion table with per-intent metrics and paired failure examples.
  • Explicit timbre-vs-structure confound test and hard-constraint checks.
  • Resource/dependency cost report and reproducible artifacts (json + human-readable summary).
  • Clear ADOPT FOR FURTHER RESEARCH / REJECT / INCONCLUSIVE decision and trigger criteria for reopening.

Adoption gate: demonstrate a reproducible, practically meaningful improvement on held-out, musically judged queries (not just classification accuracy or in-sample cosine similarity) without new feasibility violations or unacceptable timbre sensitivity. If no such improvement appears, close with the negative evidence and keep symbolic baselines unchanged.

Non-goals

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions