Skip to content

Default duckdb-bench to the V2 scan for TPC-DS and ClickBench - #11047

Merged
joseph-isaacs merged 2 commits into
developfrom
ji/duckdb-scan-v2-tpcds-clickbench
Oct 30, 2026
Merged

joseph-isaacs merged 2 commits into
developfrom
ji/duckdb-scan-v2-tpcds-clickbench

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Switches duckdb-bench to the planner-driven V2 scan by default for TPC-DS and ClickBench (TPC-H already uses it). Pass --scan-v2 false or set VORTEX_SCAN_V2=0 to use V1.

Wrong results fix

With V2, TPC-DS Q61, Q93 and Q98 returned different results than Parquet and V1. The DuckDB exporter's copy path in set_validity_zero_copy passed the bit offset within the export to copy_from_slice but ignored the BitBuffer's own bit offset. A validity bitmap that starts at a nonzero bit offset was therefore exported shifted, moving nulls by a few rows. V2 produces such bitmaps often, but V1 can hit the same bug. Regression test: test_set_validity_values_with_bit_buffer_offset.

DuckDB V2 driver changes

  • Queued parts per file. A V2 file used to report work to DuckDB until its root planner was exhausted. Every arriving thread joined the current file and blocked on the root mutex, which was held across pruning I/O, while other files sat idle. ClickBench Q01 scaled to 20ms at 32 threads, compared with 10ms on V1. Now the thread that initializes a file prunes it and queues its parts, as V1 builds its splits, so spare threads move on to the next file.
  • Parts follow layout splits. Parts were fixed 65,536-row blocks that cut layout chunks in half. Two threads then exported the same dictionary, and DuckDB evaluated expressions such as REGEXP_REPLACE over it twice. ClickBench Q28 used 22% more CPU than V1. Parts now use ScanBuilder::full_file_splits, the boundaries V1 uses.
  • Pruning twice. The scan is pruned as a whole up front, and each part is pruned again when a thread takes it. A dynamic filter, such as the bound a Top-N pushes down, then skips more of the parts taken later. The second pruning reuses the zone maps the first read (pruning_again_reuses_the_first_reads).
  • No flattening of chunked batches. A V2 batch can join several layout chunks into a ChunkedArray, which convert_result canonicalized, decoding their dictionaries. Each chunk is now exported on its own.

Fix wrong DuckDB results when a validity bitmap starts at a bit offset:
the copy path ignored the bit buffer's own offset, shifting nulls. V2
scans produce such bitmaps often (TPC-DS Q61, Q93 and Q98 were wrong).

Make the DuckDB V2 driver match V1's work distribution:
- The thread that initializes a file prunes it and queues its parts, so a
  file takes threads only while parts are left and spare threads move on
  to the next file instead of waiting on the file's root.
- Cut parts on the file's layout splits, as V1 does, so one thread
  exports each layout chunk and DuckDB evaluates expressions over its
  dictionary once.
- Prune each part again when it is taken, so a dynamic filter such as a
  top-N bound skips more of the parts taken later. The second pruning
  reuses the zone maps the first read.
- Export each chunk of a chunked V2 batch on its own instead of
  canonicalizing it, which decoded its dictionaries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs joseph-isaacs added changelog/performance A performance improvement action/bench-sql Run SQL benchmarks without Vortex Compact on this PR labels Oct 30, 2026
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 30, 2026
@codspeed

codspeed Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 13.02%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 1 improved benchmark
✅ 2317 untouched benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime threshold_avx2[80] 2.9 µs 2.6 µs +13.02%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/duckdb-scan-v2-tpcds-clickbench (4f57f63) with develop (89c5009)

Open in CodSpeed

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 4ace52d clickbench-nvme 1 Explore Profiling Data
🟢 Done 4ace52d tpch-nvme-10 1 Explore Profiling Data
🟢 Done 4ace52d polarsignals 1 Explore Profiling Data
🟢 Done 4ace52d tpch-s3 1 Explore Profiling Data
🟢 Done 4ace52d tpch-nvme 1 Explore Profiling Data
🟢 Done 4ace52d fineweb 1 Explore Profiling Data
🟢 Done 4ace52d tpcds-nvme 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark PolarSignals Profiling (pr) failed! Check the workflow run for details.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark FineWeb NVMe (pr) failed! Check the workflow run for details.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-H SF=1 on NVME (pr) failed! Check the workflow run for details.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-H SF=10 on NVME (pr) failed! Check the workflow run for details.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-DS SF=1 on NVME (pr) failed! Check the workflow run for details.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark Clickbench on NVME (pr) failed! Check the workflow run for details.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-H SF=1 on S3 (pr) failed! Check the workflow run for details.

//! The root cuts the scanned rows into parts of whole 65,536-row blocks, a few per thread, and
//! spawns a chunk planner per part. A filtered scan's chunks are up to 1,048,576 rows, so without
//! parts a file of a few million rows would keep only a few threads busy.
//! DuckDB's own threads run all of it. Each file is prepared once, and the thread that initializes

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once again I think we don't need these verbose comments

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how would you like this flow explained?

@joseph-isaacs
joseph-isaacs requested a review from myrrc October 30, 2026 16:48
The scan-wide pruning ran in reader_initialize, under the file's lock, so
on selective queries over few files it sat on the critical path. It only
pays off for a dynamic filter, whose bound tightens while parts are taken.
A static filter now prunes each part on the thread that takes it, so a
file's pruning runs in parallel. A dynamic filter still prunes the scan
first and each part again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs joseph-isaacs added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 30, 2026
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Oct 30, 2026
@joseph-isaacs
joseph-isaacs merged commit fad0b7e into develop Oct 30, 2026
103 of 111 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/duckdb-scan-v2-tpcds-clickbench branch October 30, 2026 16:58
@joseph-isaacs joseph-isaacs mentioned this pull request Nov 9, 2026
16 of 46 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants