Skip to content

Benchmark public PCO selections and cold/retained probe batches - #10824

Closed
danking wants to merge 11 commits into
developfrom
dk/pco-selection-benchmarks
Closed

danking wants to merge 11 commits into
developfrom
dk/pco-selection-benchmarks

Conversation

@danking

@danking danking commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Add public PCO benchmarks for decoding, metadata opening, take, slice, range selection, and scalar probes, using fixed workloads and untimed correctness checks. These establish a comparable baseline for the page-index optimizations using the existing benchmark and CodSpeed infrastructure.

Signed-off-by: Daniel King <dan@spiraldb.com>
… timing

Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
@codspeed

codspeed Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 11.23%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 1 improved benchmark
✅ 2317 untouched benchmarks
🆕 26 new benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime threshold_avx2[80] 3 µs 2.7 µs +11.23%
🆕 CPU Simulation first_probes[defined] N/A 120.4 ms N/A
🆕 CPU Simulation first_probes[missing] N/A 120.3 ms N/A
🆕 CPU Simulation full[non_nullable] N/A 354.8 ms N/A
🆕 CPU Simulation full[nullable] N/A 678.9 ms N/A
🆕 CPU Simulation metadata_open_batch[non_nullable] N/A 478.1 ms N/A
🆕 CPU Simulation metadata_open_batch[nullable] N/A 427.4 ms N/A
🆕 CPU Simulation retained_probes[non_nullable] N/A 346.8 ms N/A
🆕 CPU Simulation retained_probes[nullable] N/A 463.1 ms N/A
🆕 CPU Simulation selection[("broad", "non_nullable")] N/A 391.3 ms N/A
🆕 CPU Simulation selection[("broad", "nullable")] N/A 715.6 ms N/A
🆕 CPU Simulation selection[("contiguous", "non_nullable")] N/A 453.3 ms N/A
🆕 CPU Simulation selection[("contiguous", "nullable")] N/A 777.4 ms N/A
🆕 CPU Simulation selection[("mixed", "non_nullable")] N/A 391.3 ms N/A
🆕 CPU Simulation selection[("mixed", "nullable")] N/A 715.6 ms N/A
🆕 CPU Simulation selection[("narrow_repeated", "non_nullable")] N/A 70.5 ms N/A
🆕 CPU Simulation selection[("narrow_repeated", "nullable")] N/A 71.1 ms N/A
🆕 CPU Simulation selection[("near_full", "non_nullable")] N/A 355.6 ms N/A
🆕 CPU Simulation selection[("near_full", "nullable")] N/A 679.5 ms N/A
🆕 CPU Simulation selection[("ranges", "non_nullable")] N/A 354.7 ms N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing dk/pco-selection-benchmarks (e599953) with develop (da9a880)

Open in CodSpeed

@danking danking added the changelog/chore A trivial change label Oct 26, 2026
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking

danking commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

@claude review

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @danking's task in 59s —— View job


Review: PCO selection benchmarks

This adds benchmarks only, with no production code. The oracle checks before each timed loop are good: every workload is checked for correctness against the uncompressed PrimitiveArray before it is timed. I found no correctness bugs. Most of the feedback is about how much CI time these cost and how readable the reports are.

Should fix before un-drafting

  1. The workloads are about 10× over the stated budget. The PR body aims for "tens of milliseconds" per operation. CodSpeed reports much more:

    Benchmark CodSpeed time per op
    full[true] 540 ms
    selection[("broad", true)] 577 ms
    retained_probes[true] 463 ms
    metadata_open_batch[false] 385 ms
    first_probes[*] 122 ms

    That is for 40 new cases in the shared encoding shard, and the CPU-simulation shard runs under valgrind, so wall-clock time is much higher still. Consider lowering ROWS from 4M to 1M and PROBES from 1M to about 64K. Per-probe numbers come from dividing by the count anyway, so the reporting stays the same.

  2. Fixtures are rebuilt and re-checked for every case (compressed() / input(), selection.rs:45-63).

    • Every case of every bench function compresses 4M rows from scratch.
    • Several cases also build input() two or three times: in compressed, in reference, and in the oracle.
    • None of this is timed, but under CodSpeed simulation it dominates the job's wall time.
    • Fix: cache them once per nullability, e.g. static FIXTURES: LazyLock<[(PrimitiveArray, ArrayRef); 2]>. Then call LazyLock::force in main like scalar.rs does.

    Fix this →

  3. CodSpeed says one benchmark measured no execution time. It counts as unchanged, so it currently guards nothing. Find out which case it is in the "optimized_out" filter on CodSpeed. Then either make its result observable (black_box the input as well as the output) or drop the case.

Smaller suggestions

  1. The _ => arm hides mistakes (selection.rs:149). It silently uses [0] as the indices. For near_full that value is never used, but a typo in SELECTIONS would turn into a 1-row take that still passes its oracle check.

    • Better: handle near_full explicitly (e.g. Option<ArrayRef>) and make the fallback unreachable!("unknown selection {kind}").
    • Also, selection takes a slice path for near_full and wraps a view for narrow_repeated, picked by kind == string comparisons in three places. An enum Selection with Display would be clearer and would make bad cases impossible to write.
  2. The bool args are hard to read in reports.

    • full[true], first_probes[false] and metadata_open_batch[true] don't say whether true means nullable or missing.
    • In first_probes the bool means missing; everywhere else it means nullable. The two are easy to mix up on CodSpeed.
    • Named string args, or an enum with Display ("nullable" / "non_nullable", "defined" / "missing"), would make the dashboard readable.
  3. Missing global allocator. scalar.rs and the other encoding benches install #[global_allocator] static GLOBAL: MiMalloc, and mimalloc is already a dev-dependency. Without it these numbers use the system allocator. That matters most for metadata_open_batch and write, which allocate on every call.

  4. retained_probes overlaps scalar.rs.

    • Every one of its probes falls in 8192..8448. With values_per_page = 1024 that is a single page, so this is the same "clustered within one page" case scalar.rs already measures, just with 1000× more iterations.
    • If the intent is to measure keeping decoded state across pages, add a case that crosses pages.
    • If not, update the module doc to say this is single-page.
  5. metadata_open_batch seems to decode everything. It reports about the same time as full[false] (385 ms vs 344 ms). That suggests deserialization currently decodes or validates all pages, which is probably the very thing a later PR in the series will improve. Worth confirming that is expected, because the comment at selection.rs:268 says this "measures metadata/index preparation".

  6. Nit: the PR body is missing some spaces: "are256", "are1,048,576", "below10ms".

Unrelated CodSpeed regressions

The six regressions in the CodSpeed comment are in vortex-array, runend and the Python filter benches. This PR does not touch them, and CodSpeed warns that base and head ran on different runner types. They look like noise.

Checks

I only read the code. I did not run clippy, fmt or the benches; your CLAUDE.md leaves checks to you, and the PR already notes that scoped compile and lint are pending.

Signed-off-by: Daniel King <dan@spiraldb.com>
…load

Signed-off-by: Daniel King <dan@spiraldb.com>
@danking

danking commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

@claude review

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @danking's task in 51s —— View job


Review: PCO selection benchmarks

  • Gather context
  • Read encodings/pco/benches/selection.rs and Cargo.toml
  • Check what changed since the last review
  • Post review

Nothing has changed since the last review. HEAD is still 2736d4c, the commit both the CodSpeed report and my earlier review were based on. Most of that review still applies: cache the fixtures in a LazyLock, add the MiMalloc global allocator like scalar.rs, replace the _ => fallback with an enum, give the bool args readable names, and note that retained_probes reads a single page (rows 8192..8448 with 1024-row pages). This pass corrects two points from that review and adds a few smaller ones. I still found no correctness bugs, and the untimed oracle checks are solid.

Corrections to the earlier review

  1. Point 8 (metadata_open_batch "decodes everything") was wrong. The 385 ms covers OPENS = 256 deserializations, which is about 1.5 ms per open. It is close to full[false] only by coincidence, so it does not show that deserialization decodes every page. The comment at selection.rs:269 is fine as written. If you want reports in per-open time, divide by OPENS the same way the module doc does for probes.
  2. Point 1 (budget) needs a caveat. CodSpeed's "CPU Simulation" numbers are simulated instruction cost, not wall time. They don't directly disprove the "tens of milliseconds on the baseline" target, which is a wall-time claim. The CI-time concern still holds, though. Each case compresses 4M rows outside the timer, with 40 cases, under valgrind. So caching the fixtures matters more than shrinking ROWS. Wall-time calibration on the target runner is still pending, as the PR body says.
  3. Point 3: CodSpeed reports 2 benchmarks with no measured execution time, not 1. The report doesn't say whether they come from this PR. Check the optimized_out filter before treating it as a blocker.

New, smaller points

  1. Magic numbers (selection.rs:128,132,137,140):
    • 4_194_304u64 repeats ROWS; use ROWS as u64.
    • 524288 appears three times; a const TAKE_LEN would make the shared workload size explicit.
    • The stride 32749 appears three times. It is odd, so it gives distinct indices mod 2²², which is worth a short comment and a named const.
  2. contiguous and near_full are a good pair. Both select the same rows, 1..ROWS-1: one through take, the other through slice. A short comment would stop someone from later "deduplicating" them as redundant.
  3. drop(actual) (selection.rs:252) does nothing useful. It drops a single Scalar that holds no meaningful untimed state, so it can go. The drop(check) / drop(reference) calls before the timed sections are reasonable.
  4. first_probes uses rows 8190 (null) and 8191 (valid). Both sit in page 7, and 8191 is the last row of that page. That's fine, but it ties into point 7 of the earlier review: retained probes use page 8 and first probes use page 7. If later PRs change page-boundary handling, a first probe that lands mid-page would be a less boundary-sensitive control.
  5. Timed take construction is intentional and fine. selection times both building the lazy take / slice and executing it. indices.clone() is just an Arc bump, and divan drops outputs outside the timer.

Checks

I only read the code. I did not run clippy, fmt or the benches; CLAUDE.md leaves verification to you, and the PR notes that scoped compile, lint and cloud calibration are still pending.

@danking
danking added this pull request to stack #10902 October 27, 2026 20:13
Signed-off-by: Daniel King <dan@spiraldb.com>

danking commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the benchmark review in e599953: cache each primitive/PCO fixture once with LazyLock and force setup before the runner, use the existing MiMalloc dependency, label nullable/non-nullable and defined/missing cases, name the workload constants, reject unknown selection names, and remove the unnecessary scalar drop. The retained case is explicitly documented as single-page; scalar.rs supplies the scattered control.

The fixed workload sizes are unchanged. CPU Simulation numbers are not wall-time calibration, and the allocator/setup changes have not been rebenchmarked. The repository-wide optimized-out warning was not attributed to these cases, so no benchmark was removed or disabled. Local independent source review passed; no builds, tests, formatting commands, or benchmarks were run.

@github-actions

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Nov 15, 2026
@danking danking closed this Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change stale This PR is stale and will be auto-closed soon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant