Skip to content

Run TPC-DS SF=10 on NVMe in the PR SQL benchmarks - #11194

Merged
joseph-isaacs merged 1 commit into
developfrom
ji/sleepy-rubin-2w1aci
Nov 4, 2026
Merged

joseph-isaacs merged 1 commit into
developfrom
ji/sleepy-rubin-2w1aci

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

action/bench-sql now runs TPC-DS SF=10 on NVMe, on DataFusion only (Parquet and Vortex). Nightly still runs all four targets, adding DuckDB on Parquet and on Vortex.

DuckDB is left out of the PR runs because DuckDB on Vortex ran out of memory on q64 in 3 of the last 4 nightly runs (could not allocate block of size 256.0 KiB (150.6 GiB/150.6 GiB used)).

Expected time, from the nightly job (same machine, same iteration count, all four targets):

Nightly run Job total Data gen Benchmark Result
41746630649 7.9 min 40 s 6.5 min ✅
41552682909 12.1 min 40 s 10.7 min ❌ q64 OOM
41367242730 12.0 min 40 s 10.7 min ❌ q64 OOM
40980135943 12.5 min 40 s 11.1 min ❌ q64 OOM

In the passing run, the two DataFusion targets took 1.8 min (Parquet) and 1.7 min (Vortex). With data generation and setup, the PR job should take about 4–5 minutes. That is less than existing pr jobs such as TPC-H SF=1 on S3 (about 11 min), so the overall run should not take longer.

Changes

  • ci_matrix/catalog.py: adds a DATAFUSION coverage (DataFusion on Parquet and Vortex). tpcds-nvme-10 uses it in the pr and pr-full presets and keeps DEFAULT for nightly. It is in pr-full because a test requires every pr benchmark to be in pr-full. The benchmark run on every commit to develop does not change.
  • tests/test_matrix.py: updates the expected benchmark ids for each preset, and checks that tpcds-nvme-10 runs only the DataFusion targets in pr and pr-full.
  • docs/developer-guide/benchmarking.md: the action/bench-sql description now lists TPC-DS SF=10 on NVMe (DataFusion only).

Checks: pytest bench-orchestrator/tests/test_matrix.py (15 passed), ruff format --check and ruff check on bench-orchestrator.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UeniBBdh1yBcuLEcJ1vAUY

Add tpcds-nvme-10 to the pr preset (action/bench-sql) at the default
DataFusion/DuckDB x Parquet/Vortex targets, and to pr-full so pr stays a
subset of it. Nightly keeps running it; develop does not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBBdh1yBcuLEcJ1vAUY

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs merged commit 4724b9e into develop Nov 4, 2026
99 of 100 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/sleepy-rubin-2w1aci branch November 4, 2026 10:22

This branch was successfully deployed

1 active deployment
vercel-docs — 3a23f8cb Deployed Nov 4, 2026 by joseph-isaacs via Publish preview / Deploy #8082
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants