Repository navigation
Benchmarks write and record zstd Parquet, and TPC-DS SF1 queries Snappy Parquet and Lance - #11720
Open
lwwmanning wants to merge 9 commits into
Open
lwwmanning wants to merge 9 commits into
lwwmanning wants to merge 9 commits into
Conversation
capture-file-sizes.py reads each Parquet file's footer and names its format by codec: zstd stays `parquet`, the codec the benchmarks write, and other codecs record as `parquet-<codec>`, as `vortex` and `vortex-compact` name Vortex's. ClickBench and FineWeb measure downloaded files, and the statpopgen and vortex suites write without zstd, so their sizes were recorded as zstd Parquet. Signed-off-by: Will Manning <will@willmanning.io>
compress-bench writes each dataset once more as Snappy Parquet, untimed, and records its size as `parquet-snappy` beside `parquet` (zstd level 3). Only sizes: Snappy adds no compression or decompression timings. Signed-off-by: Will Manning <will@willmanning.io>
compress-bench now writes each dataset once more with the compact Vortex strategy and records its size as vortex-compact, as the SQL benchmarks name it. The write is not timed and does not run in GPU mode. bench can then compare Vortex and compact Vortex on the compression datasets, as it does on the SQL benchmarks' files. Signed-off-by: Will Manning <will@willmanning.io>
Develop's TPC-DS SF1 run now writes three formats that no query reads, after its queries, so their file sizes are recorded with the others: - `data-gen --size-formats arrow-ipc,parquet-snappy` writes uncompressed Arrow IPC and Snappy Parquet from the base Parquet files. The Snappy files keep the base files' other writer settings and row groups. - `lance-bench --data-only` writes the Lance data and exits. - `vx-bench write-size-formats` runs both, from a new `size_formats` list on a matrix coverage. CI runs it before "Capture file sizes". `capture-file-sizes.py` records the `parquet-snappy` directory, and labels its files by codec like the `parquet` directory's. At SF1 the writes take about 8 seconds and 3.8 GB of disk. Signed-off-by: Will Manning <will@willmanning.io>
Develop's TPC-DS SF1 run now queries Snappy Parquet on DataFusion and DuckDB, and Lance on DataFusion. data-gen writes Snappy Parquet as the parquet-snappy format, from the base Parquet files with the same row groups. Arrow IPC stays a size-only format. Lance skips TPC-DS q72, which overflows its Utf8 offsets. bench ranks a series that misses a query below every series that ran them all. Signed-off-by: Will Manning <will@willmanning.io>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Signed-off-by: Will Manning <will@willmanning.io>
… vortex suite Every benchmark writes zstd level 3 Parquet, but three writers did not: statpopgen and the SpatialBench zone rewrite passed no writer settings, so parquet-rs wrote them uncompressed, and the vortex suite's DuckDB COPY named no codec, so DuckDB wrote Snappy. Their Parquet sizes and times measured a different codec from every other suite. All three now write zstd level 3. statpopgen's CI data cache also keys on the shared writer settings, so a change to them regenerates the cached files. Signed-off-by: Will Manning <will@willmanning.io>
`parquet-snappy` is now Parquet as DuckDB writes it: `COPY ... (FORMAT parquet, COMPRESSION snappy)` in the `duckdb` CLI, with DuckDB's row groups, dictionaries, statistics, and Bloom filters. data-gen writes the query files this way, and compress-bench records its Snappy size from the same writer. Every benchmark runner already installs the CLI. File sizes are named by directory again, not by codec. `parquet` is the suite's base Parquet: zstd level 3 where the benchmarks write it, and the dataset's own files where the suite prescribes them, as ClickBench's Snappy files are. This keeps ClickBench's `parquet` series continuous and keeps `parquet-snappy` for DuckDB's files. Signed-off-by: Will Manning <will@willmanning.io>
#11676 changed the integer cast's overflow error from "value exceeds target range" to "values exceed target range". The device batch test looked for the old wording, so CUDA tests fail on develop's code. Match the words both share. Signed-off-by: Will Manning <will@willmanning.io>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bench.vortex.dev compares file sizes with zstd Parquet. Every benchmark now writes zstd level 3 unless the suite prescribes its own files. The statpopgen suite and SpatialBench's zone table wrote uncompressed Parquet, and the
vortexsuite wrote Snappy. Now all three write zstd. ClickBench and FineWeb still query their published files asparquet.A new format,
parquet-snappy, is Parquet as DuckDB writes it by default. The DuckDB CLI writes it from the base Parquet. compress-bench records its size and the compact Vortex size. On develop, TPC-DS at scale factor 1 (SF1) queries it on DataFusion and DuckDB, and Lance on DataFusion. It also writes Arrow IPC for its size. Lance skips q72, which overflows its 32-bit string offsets.Behavior change: Parquet sizes and times for statpopgen, SpatialBench, and the
vortexsuite change at this commit. The bench side is vortex-data/vortex.dev#227.🤖 Generated with Claude Code