Repository navigation
Cut CodSpeed time in fsst_take, split_collection and OnPair decode - #11470
Conversation
Both benchmarks spent most of their CodSpeed time outside the measured code, building fixtures under the CPU simulation. - fsst_take: the fixture grew its rows by 2% per step from an uncompressed estimate and retrained the FSST compressor at every step, 26 to 81 times per case. It now scales the row length by the compression ratio it just observed, which reaches the target in 2 to 4 compressions. Every fixture still reaches its compressed-size target, and the chosen row lengths stay within about 2% of the previous ones. - split_collection: the files were written with the BtrBlocks compressor, but split collection walks only the layout tree. They are now written uncompressed. The layout trees and the collected splits are identical to before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAqXtmDDNUP9pHsfSo Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The three `(UrlLog, 1_000_000)` cases cost 2.2 to 2.9 s each in the CodSpeed simulation, which is 10.5 s of the binary's time. They measure the same kernels as the `(UrlLog, 100_000)` cases at ten times the size. They now run only locally, as `split_collection` already does for its scaling cases. CodSpeed measures 9 of the 12 benchmarks, and the shard counts are updated to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAqXtmDDNUP9pHsfSo Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will degrade performance by 13.4%2 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Summary
In the latest
developCodSpeed run (43502927659), these benchmarks spent most of their time either building fixtures under the CPU simulation or measuring cases far larger than the rest of their sweep. This PR removes that time without changing what the remaining benchmarks measure.Changes
vortex-fsst/fsst_take(8.5 s in CI)vortex-file/split_collectionSplitBy::LayoutandSplitBy::default()are identical to before.vortex-onpair/decode(10.5 s in CI)(UrlLog, 1_000_000)cases cost 2.2–2.9 s each. They measure the same kernels as the(UrlLog, 100_000)cases at ten times the size.#[cfg(not(codspeed))]), assplit_collectionalready does for its scaling cases.scripts/codspeed_bench_counts.jsonis updated to match. These three benchmarks lose their CodSpeed history.Left alone on purpose:
vortex-spatial/binary_predicates(4.5 s): locally each benchmark after the first takes about 0.01 s. Its CI cost is the measured iteration under instrumentation, which the file sizes deliberately against its per-iteration budget.vortex-arrow/to_arrow(29 s over 130 benchmarks): locally each case takes 0.02–0.04 s including setup, so its CI cost is also mostly measured work. Sharing fixtures between its benchmarks could let one benchmark warm cached statistics for the next.Measurements
Local
valgrind --tool=callgrindruns of the CodSpeed builds (--cfg codspeed,VORTEX_BENCH_VARIANT=simulation). These include setup but not CodSpeed's instrumentation, so they show the setup savings rather than CI times.fsst_takefsst_take, the 18takecasessplit_collectionwarm(first to build the file)onpair/decodeChecks
--cfg codspeed: no warnings. CodSpeed now runs 9decode, 24fsst_takeand 8split_collectionbenchmarks.fsst_takecase, and compared trees and splits for everysplit_collectionfile with a temporary debugmain.cargo +nightly-2026-09-10 fmt --checkon the changed bench files.pytest scripts/tests/test_codspeed_shards.py(22 passed).Generated by Claude Code