Skip to content

Cut CodSpeed time in fsst_take, split_collection and OnPair decode - #11470

Merged
joseph-isaacs merged 2 commits into
developfrom
ji/codspeed-bench-setup
Nov 15, 2026
Merged

joseph-isaacs merged 2 commits into
developfrom
ji/codspeed-bench-setup

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Summary

In the latest develop CodSpeed run (43502927659), these benchmarks spent most of their time either building fixtures under the CPU simulation or measuring cases far larger than the rest of their sweep. This PR removes that time without changing what the remaining benchmarks measure.

Changes

  • vortex-fsst/fsst_take (8.5 s in CI)
    • The fixture searched for a row length that reaches its compressed-size target. It started from an uncompressed estimate and grew the rows 2% at a time, retraining the FSST compressor at every step: 26 to 81 times per case.
    • It now scales the row length by the compression ratio it just observed, which reaches the target in 2 to 4 compressions.
    • Every fixture still reaches its target, which is what the compressed-bytes gate under test depends on. The chosen row lengths stay within about 2% of the previous ones, and the codes sizes within about 3%, so expect a small one-time shift on CodSpeed.
  • vortex-file/split_collection
    • The files were written with the BtrBlocks compressor, but split collection walks only the layout tree. They are now written uncompressed.
    • For every config and the misaligned file, the layout trees and the splits from both SplitBy::Layout and SplitBy::default() are identical to before.
  • vortex-onpair/decode (10.5 s in CI)
    • The three (UrlLog, 1_000_000) cases cost 2.2–2.9 s each. They measure the same kernels as the (UrlLog, 100_000) cases at ten times the size.
    • They now run only locally (#[cfg(not(codspeed))]), as split_collection already does for its scaling cases.
    • CodSpeed measures 9 of the 12 benchmarks, and scripts/codspeed_bench_counts.json is updated to match. These three benchmarks lose their CodSpeed history.

Left alone on purpose:

  • vortex-spatial/binary_predicates (4.5 s): locally each benchmark after the first takes about 0.01 s. Its CI cost is the measured iteration under instrumentation, which the file sizes deliberately against its per-iteration budget.
  • vortex-arrow/to_arrow (29 s over 130 benchmarks): locally each case takes 0.02–0.04 s including setup, so its CI cost is also mostly measured work. Sharing fixtures between its benchmarks could let one benchmark warm cached statistics for the next.

Measurements

Local valgrind --tool=callgrind runs of the CodSpeed builds (--cfg codspeed, VORTEX_BENCH_VARIANT=simulation). These include setup but not CodSpeed's instrumentation, so they show the setup savings rather than CI times.

Binary Before After
fsst_take 1.84 s 0.65 s
fsst_take, the 18 take cases 1.14 s 0.27 s
split_collection warm (first to build the file) 0.11 s 0.02 s
onpair/decode 2.07 s 0.78 s

Checks

  • Built the three benches with --cfg codspeed: no warnings. CodSpeed now runs 9 decode, 24 fsst_take and 8 split_collection benchmarks.
  • Compared both fixture searches for every fsst_take case, and compared trees and splits for every split_collection file with a temporary debug main.
  • cargo +nightly-2026-09-10 fmt --check on the changed bench files.
  • pytest scripts/tests/test_codspeed_shards.py (22 passed).
  • I did not run clippy or the crate test suites.

Generated by Claude Code

joseph-isaacs and others added 2 commits November 13, 2026 21:47
Both benchmarks spent most of their CodSpeed time outside the measured
code, building fixtures under the CPU simulation.

- fsst_take: the fixture grew its rows by 2% per step from an
  uncompressed estimate and retrained the FSST compressor at every step,
  26 to 81 times per case. It now scales the row length by the
  compression ratio it just observed, which reaches the target in 2 to 4
  compressions. Every fixture still reaches its compressed-size target,
  and the chosen row lengths stay within about 2% of the previous ones.
- split_collection: the files were written with the BtrBlocks
  compressor, but split collection walks only the layout tree. They are
  now written uncompressed. The layout trees and the collected splits
  are identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBAqXtmDDNUP9pHsfSo
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The three `(UrlLog, 1_000_000)` cases cost 2.2 to 2.9 s each in the
CodSpeed simulation, which is 10.5 s of the binary's time. They measure
the same kernels as the `(UrlLog, 100_000)` cases at ten times the size.
They now run only locally, as `split_collection` already does for its
scaling cases. CodSpeed measures 9 of the 12 benchmarks, and the shard
counts are updated to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBAqXtmDDNUP9pHsfSo
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs enabled auto-merge (squash) November 13, 2026 22:05
@codspeed

codspeed Bot commented Nov 13, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 13.4%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

❌ 2 regressed benchmarks
✅ 1931 untouched benchmarks
⏩ 53 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation take[(16, 8, 1)] 103.9 µs 124.7 µs -16.66%
❌ CPU Simulation take[(16, 8, 3)] 119.5 µs 132.8 µs -10.01%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/codspeed-bench-setup (c6acdf4) with develop (d7aa24e)

Open in CodSpeed

Footnotes

  1. 53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@robert3005 robert3005 added the changelog/chore A trivial change label Nov 15, 2026
@joseph-isaacs
joseph-isaacs merged commit d7cbd6f into develop Nov 15, 2026
100 of 103 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/codspeed-bench-setup branch November 15, 2026 15:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants