Skip to content

[Parity #478] Track dtype, memory layout, preprocessing and conversion work explicitly #502

Open

Description

@godofecht

Parent: #478.

Flow/sklearn timings must expose representation differences rather than accidentally rewarding one side for cheaper inputs. Record source dtype, estimator internal dtype, row/column-major layout, contiguity/alignment, preprocessing/scaling, label encoding and every timed/untimed conversion or copy.

Where sklearn internally promotes or copies, distinguish API-realistic end-to-end timing from kernel-only equivalent-representation timing. Where Flow uses f32 and sklearn f64, make that a visible semantic/performance dimension with numerical-error evidence.

Acceptance: canonical and scaled artifacts contain representation/preprocessing metadata; hidden untimed conversion bytes = 0; equivalent-representation microbenchmarks exist for disputed gaps; Pages can distinguish end-to-end API wins from dtype/layout advantages.

Activity

  1. godofecht commented on Aug 25, 2026

    @godofecht
    OwnerAuthor

    I worked through the representation-parity part of this issue and archived the full derivation + Flow experiment in OpenProjects as ml/representation-normalized-benchmark-certificates.md and .flow.

    The key result is a non-identifiability theorem: if two implementations enter their timed kernels with different representations, end-to-end timing alone cannot identify which numerical kernel is faster. Example: K_A=12, K_B=10, but conversion costs C_A=0, C_B=5 give T_A=12 < 15=T_B; the product winner has the slower kernel.

    I therefore split eligibility into two certificates. API/product timing is valid when semantic parity holds and hidden untimed conversion bytes are zero: E_product = S && H==0. Kernel/implementation timing additionally requires equal kernel-entry representation signatures: E_kernel = S && H==0 && (R_A == R_B), with R=(dtype, layout, alignment, endianness, preprocessing, shape).

    For dense materializing conversion, there is also an exact traffic lower bound: B_min >= N*(b_source+b_target). On Digits (1797*64=115008 elements), f64→f32 requires at least 1,380,096 bytes of one-pass read/write traffic. This should be emitted in a conversion ledger rather than disappearing into a benchmark boundary.

    Concrete direction: every canonical operation should record public-source and kernel-entry representation signatures plus (copies, bytes_read, bytes_written, hidden_untimed_bytes). Keep API-realistic rows with all required conversion work timed, and add equivalent-representation microbenchmarks where kernel attribution is needed. A published row should never have hidden untimed conversion bytes > 0.

    No hosted CI/workflows were touched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions