Skip to content

[Bench #478] Benchmark every Flow-implemented estimator operation, not only the 19 headline rows #497

Open

Description

@godofecht

Parent: #478

The architecture inventory contains 491 sklearn estimator-operation rows, while the canonical headline intentionally covers only 19 estimator/dataset combinations. Any Flow-implemented estimator or operation outside that set is currently capable of hiding a severe regression.

Build an implementation-surface registry and require every public Flow estimator operation to have at least correctness/parity smoke, representative warm timing, scaled timing classification, memory evidence tier and execution-substrate attribution. Unsupported sklearn operations should be distinguished from implemented-but-unbenchmarked operations.

Validation is local-only. Do not add, enable, trigger, inspect, wait for, or rely on GitHub Actions or other hosted CI/workflows. Coverage ratcheting should be enforced by a deterministic local command from a clean checkout and recorded in machine-readable artifacts keyed to the exact source SHA/environment.

Acceptance: implemented public operations with no benchmark evidence = 0; coverage is machine-readable and locally ratcheted; the 19-row suite remains the stable headline subset, while the full surface produces a broader wins/ties/losses dashboard and new losses become focused issues.

Activity

  1. godofecht commented on Aug 28, 2026

    @godofecht
    OwnerAuthor

    I worked through the implementation-surface registry design as a concrete coverage ratchet.

    The useful split is public-operation evidence versus reusable implementation-level evidence. Let S be public implemented estimator operations and φ:S→I map each operation to a semantic implementation-equivalence class. Each public op keeps its own correctness/parity smoke, while expensive warm/scaled/memory/substrate evidence is keyed to φ(o). Coverage is

    Covered(o) = Smoke(o) && ((H[φ(o)] & R[o]) == R[o]).

    This gives two properties we want simultaneously: a new public API operation cannot hide because it must get operation-local smoke evidence, while aliases/wrappers that truly lower to the same numerical/representation/substrate contract do not need duplicate heavy benchmark records.

    Under the rule that heavy evidence may only be reused within a valid semantic implementation class, one heavy evidence record per represented implementation class is necessary and sufficient. Evidence growth is monotone: adding evidence cannot make an already-covered operation uncovered; only API growth, implementation-identity changes, or stronger evidence requirements can trip the ratchet.

    I encoded the model and the local validator cases in Flow and archived the full derivation in OpenProjects:

    • ml/implementation-surface-benchmark-ratchet.md
    • ml/implementation-surface-benchmark-ratchet.flow

    The fixed experiment has 8 public operations backed by 5 implementation classes, proves an added alias can reuse heavy evidence after its own smoke passes, proves a genuinely new implementation fails coverage until a new evidence record exists, and proves requirement strengthening fans out exactly to users of the deficient implementation class. I also independently exhaustively checked monotonicity over the complete 4-bit evidence lattice locally.

    A production implementation_surface.json can therefore remain operation-complete without turning the 19-row headline benchmark into the coverage mechanism. The next useful measurement is |S|/|I| on the real Flow API: that tells us how much expensive benchmark work can soundly collapse through shared implementation identities.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions