Skip to content

Benchmarks write and record zstd Parquet, and TPC-DS SF1 queries Snappy Parquet and Lance - #11720

Open
lwwmanning wants to merge 9 commits into
developfrom
wm/file-size-parquet-codecs
Open

lwwmanning wants to merge 9 commits into
developfrom
wm/file-size-parquet-codecs

Conversation

@lwwmanning

@lwwmanning lwwmanning commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

bench.vortex.dev compares file sizes with zstd Parquet. Every benchmark now writes zstd level 3 unless the suite prescribes its own files. The statpopgen suite and SpatialBench's zone table wrote uncompressed Parquet, and the vortex suite wrote Snappy. Now all three write zstd. ClickBench and FineWeb still query their published files as parquet.

A new format, parquet-snappy, is Parquet as DuckDB writes it by default. The DuckDB CLI writes it from the base Parquet. compress-bench records its size and the compact Vortex size. On develop, TPC-DS at scale factor 1 (SF1) queries it on DataFusion and DuckDB, and Lance on DataFusion. It also writes Arrow IPC for its size. Lance skips q72, which overflows its 32-bit string offsets.

Behavior change: Parquet sizes and times for statpopgen, SpatialBench, and the vortex suite change at this commit. The bench side is vortex-data/vortex.dev#227.

🤖 Generated with Claude Code

capture-file-sizes.py reads each Parquet file's footer and names its
format by codec: zstd stays `parquet`, the codec the benchmarks write,
and other codecs record as `parquet-<codec>`, as `vortex` and
`vortex-compact` name Vortex's. ClickBench and FineWeb measure
downloaded files, and the statpopgen and vortex suites write without
zstd, so their sizes were recorded as zstd Parquet.

Signed-off-by: Will Manning <will@willmanning.io>
compress-bench writes each dataset once more as Snappy Parquet, untimed,
and records its size as `parquet-snappy` beside `parquet` (zstd level
3). Only sizes: Snappy adds no compression or decompression timings.

Signed-off-by: Will Manning <will@willmanning.io>
compress-bench now writes each dataset once more with the compact Vortex
strategy and records its size as vortex-compact, as the SQL benchmarks
name it. The write is not timed and does not run in GPU mode. bench can
then compare Vortex and compact Vortex on the compression datasets, as it
does on the SQL benchmarks' files.

Signed-off-by: Will Manning <will@willmanning.io>
Develop's TPC-DS SF1 run now writes three formats that no query reads,
after its queries, so their file sizes are recorded with the others:

- `data-gen --size-formats arrow-ipc,parquet-snappy` writes uncompressed
  Arrow IPC and Snappy Parquet from the base Parquet files. The Snappy
  files keep the base files' other writer settings and row groups.
- `lance-bench --data-only` writes the Lance data and exits.
- `vx-bench write-size-formats` runs both, from a new `size_formats`
  list on a matrix coverage. CI runs it before "Capture file sizes".

`capture-file-sizes.py` records the `parquet-snappy` directory, and
labels its files by codec like the `parquet` directory's.

At SF1 the writes take about 8 seconds and 3.8 GB of disk.

Signed-off-by: Will Manning <will@willmanning.io>
Develop's TPC-DS SF1 run now queries Snappy Parquet on DataFusion and
DuckDB, and Lance on DataFusion. data-gen writes Snappy Parquet as the
parquet-snappy format, from the base Parquet files with the same row
groups. Arrow IPC stays a size-only format.

Lance skips TPC-DS q72, which overflows its Utf8 offsets. bench ranks a
series that misses a query below every series that ran them all.

Signed-off-by: Will Manning <will@willmanning.io>
@lwwmanning lwwmanning added the changelog/chore A trivial change label Nov 20, 2026
@lwwmanning
lwwmanning marked this pull request as ready for review November 21, 2026 01:58
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T02:37:07.357237Z 43d89ce New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Signed-off-by: Will Manning <will@willmanning.io>
… vortex suite

Every benchmark writes zstd level 3 Parquet, but three writers did not:
statpopgen and the SpatialBench zone rewrite passed no writer settings,
so parquet-rs wrote them uncompressed, and the vortex suite's DuckDB
COPY named no codec, so DuckDB wrote Snappy. Their Parquet sizes and
times measured a different codec from every other suite. All three now
write zstd level 3. statpopgen's CI data cache also keys on the shared
writer settings, so a change to them regenerates the cached files.

Signed-off-by: Will Manning <will@willmanning.io>
@lwwmanning lwwmanning changed the title Benchmarks record Parquet sizes by codec, and TPC-DS SF1 queries Snappy Parquet and Lance Benchmarks write and record zstd Parquet, and TPC-DS SF1 queries Snappy Parquet and Lance Nov 21, 2026
`parquet-snappy` is now Parquet as DuckDB writes it: `COPY ... (FORMAT
parquet, COMPRESSION snappy)` in the `duckdb` CLI, with DuckDB's row
groups, dictionaries, statistics, and Bloom filters. data-gen writes the
query files this way, and compress-bench records its Snappy size from the
same writer. Every benchmark runner already installs the CLI.

File sizes are named by directory again, not by codec. `parquet` is the
suite's base Parquet: zstd level 3 where the benchmarks write it, and the
dataset's own files where the suite prescribes them, as ClickBench's
Snappy files are. This keeps ClickBench's `parquet` series continuous and
keeps `parquet-snappy` for DuckDB's files.

Signed-off-by: Will Manning <will@willmanning.io>
#11676 changed the integer cast's overflow error from "value exceeds
target range" to "values exceed target range". The device batch test
looked for the old wording, so CUDA tests fail on develop's code. Match
the words both share.

Signed-off-by: Will Manning <will@willmanning.io>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant