Skip to content

Size the list layout split grid by row count - #10531

Merged
mhk197 merged 1 commit into
developfrom
mk/list-layout-split-rows
Oct 16, 2026
Merged

mhk197 merged 1 commit into
developfrom
mk/list-layout-split-rows

Conversation

@mhk197

@mhk197 mhk197 commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

ListReader places scan splits on a grid of outer rows, and the grid was fixed at 64 cells whatever the file size. The grid now grows with the row count: at least 64 cells, and enough that a cell covers at most about 8,192 rows. Files under about 512K rows are unchanged.

How list splits work

register_splits must report split points in outer rows, but a list's natural boundaries are its element blocks, which are cut in element rows. It can't read the offsets at that point, so it estimates:

  1. Take each element-block boundary as a fraction of all elements.
  2. Assume the same fraction of outer rows (i.e. that every list has the same length).
  3. Snap it to the nearest edge of a grid of equal cells, and drop duplicates.

Example: 800 outer rows, 4,000 elements in 5 blocks, an 8-cell grid:

element rows   0       800     1600    2400    3200    4000
               |-blk 0-|-blk 1-|-blk 2-|-blk 3-|-blk 4-|
boundary at           20%     40%     60%     80%

outer rows     0    100  200  300  400  500  600  700  800
               |----|----|----|----|----|----|----|----| 8 equal cells
snapped to               ▲    ▲         ▲    ▲
cell edge                2    3         5    6           (1.6→2, 3.2→3, 4.8→5, 6.4→6)

scan splits    |=========|====|=========|====|=========| 5 splits
               0         200  300       500  600       800
  • The grid caps how many splits a list can add (at most one per cell); a list with a single element block adds none.
  • Grid sizes are powers of two, so the edges of different lists' grids coincide when the scan merges split points across columns.
  • Known limitation: the estimate assumes uniform list lengths, so with skewed lengths splits have equal rows but unequal work. Neither benchmark dataset is skewed.

Grid size (this PR): cells = max(64, next_power_of_two(ceil(rows / 8192)))

rows cells rows per cell
100,000 (statpopgen) 64 (unchanged) ~1,560
524,288 64 8,192
1,000,000 (polarsignals) 128 (was 64) ~7,810
10,000,000 2,048 ~4,880

Results

Local runs, list layout ÷ develop (52a1d15): 3 interleaved rounds with rotating order, in-process iterations, Apple M5 Max (18 cores), warm page cache, release_debug with target-cpu=native. "grid 64" is the same list layout without this PR's reader change.

benchmark grid 64 this PR worst query (this PR)
polarsignals, DataFusion, vortex 0.94x 0.90x q4 1.06x (3.7 ms)
statpopgen, DataFusion, vortex 0.64x 0.63x q0 0.95x
statpopgen, DuckDB, vortex 0.73x 0.74x q0 0.98x
statpopgen, DataFusion, vortex-compact 0.52x q0 0.94x
statpopgen, DuckDB, vortex-compact 0.61x q0 1.01x
polarsignals, DataFusion, `vortex` (20 iterations)
query develop grid 64 this PR grid 64 ÷ develop this PR ÷ develop
q0 30.4 ms 45.5 ms 29.7 ms 1.49x 0.98x
q1 109.3 ms 115.6 ms 110.7 ms 1.06x 1.01x
q2 22.0 ms 6.5 ms 7.1 ms 0.30x 0.32x
q3 109.8 ms 117.0 ms 112.4 ms 1.07x 1.02x
q4 3.7 ms 3.9 ms 4.0 ms 1.05x 1.06x
q5 4.7 ms 4.7 ms 4.8 ms 0.99x 1.01x
q6 10.9 ms 10.6 ms 10.6 ms 0.97x 0.98x
q7 4.5 ms 4.4 ms 4.4 ms 0.98x 0.98x
q8 190.0 ms 201.6 ms 191.5 ms 1.06x 1.01x
q9 3.5 ms 3.5 ms 3.6 ms 1.00x 1.02x
geomean 0.94x 0.90x
statpopgen, DataFusion, `vortex` (5 iterations)
query develop grid 64 this PR grid 64 ÷ develop this PR ÷ develop
q0 0.9 ms 0.9 ms 0.9 ms 0.94x 0.95x
q1 26.0 ms 2.1 ms 1.9 ms 0.08x 0.07x
q2 68.1 ms 46.5 ms 46.3 ms 0.68x 0.68x
q3 244.0 ms 185.0 ms 191.4 ms 0.76x 0.78x
q4 230.5 ms 187.1 ms 180.0 ms 0.81x 0.78x
q5 152.4 ms 117.9 ms 114.6 ms 0.77x 0.75x
q6 506.5 ms 389.9 ms 385.0 ms 0.77x 0.76x
q7 182.1 ms 145.6 ms 146.3 ms 0.80x 0.80x
q8 178.9 ms 145.2 ms 146.2 ms 0.81x 0.82x
q9 173.6 ms 112.5 ms 112.8 ms 0.65x 0.65x
q10 419.7 ms 373.5 ms 364.6 ms 0.89x 0.87x
geomean 0.64x 0.63x
statpopgen, DuckDB, `vortex` (5 iterations)
query develop grid 64 this PR grid 64 ÷ develop this PR ÷ develop
q0 4.8 ms 4.7 ms 4.7 ms 0.98x 0.98x
q1 5.3 ms 4.9 ms 4.8 ms 0.91x 0.90x
q2 178.8 ms 99.0 ms 101.9 ms 0.55x 0.57x
q3 426.4 ms 325.1 ms 331.8 ms 0.76x 0.78x
q4 424.7 ms 324.5 ms 320.4 ms 0.76x 0.75x
q5 199.2 ms 136.4 ms 133.3 ms 0.68x 0.67x
q6 699.2 ms 473.0 ms 467.4 ms 0.68x 0.67x
q7 85.3 ms 54.8 ms 61.2 ms 0.64x 0.72x
q8 91.1 ms 55.6 ms 53.3 ms 0.61x 0.58x
q9 292.0 ms 233.9 ms 229.9 ms 0.80x 0.79x
q10 1060.5 ms 852.1 ms 852.6 ms 0.80x 0.80x
geomean 0.73x 0.74x
statpopgen, DataFusion, `vortex-compact` (5 iterations)
query develop this PR this PR ÷ develop
q0 0.9 ms 0.9 ms 0.94x
q1 44.0 ms 2.2 ms 0.05x
q2 91.7 ms 58.0 ms 0.63x
q3 254.5 ms 191.5 ms 0.75x
q4 253.2 ms 197.5 ms 0.78x
q5 182.4 ms 133.3 ms 0.73x
q6 521.9 ms 403.3 ms 0.77x
q7 521.5 ms 181.5 ms 0.35x
q8 521.7 ms 184.9 ms 0.35x
q9 171.8 ms 126.7 ms 0.74x
q10 444.3 ms 370.4 ms 0.83x
geomean 0.52x
statpopgen, DuckDB, `vortex-compact` (5 iterations)
query develop this PR this PR ÷ develop
q0 4.7 ms 4.8 ms 1.01x
q1 5.0 ms 4.9 ms 0.98x
q2 191.4 ms 120.4 ms 0.63x
q3 466.7 ms 344.9 ms 0.74x
q4 458.2 ms 341.1 ms 0.74x
q5 232.2 ms 152.9 ms 0.66x
q6 699.4 ms 485.3 ms 0.69x
q7 446.6 ms 95.0 ms 0.21x
q8 435.9 ms 101.5 ms 0.23x
q9 341.9 ms 254.6 ms 0.74x
q10 1090.8 ms 876.1 ms 0.80x
geomean 0.61x

File sizes (list layout ÷ develop): statpopgen vortex 1.93 → 1.74 GB (0.90x), statpopgen vortex-compact 1.003 → 0.977 GB (0.97x), polarsignals 718.8 → 714.2 MB (0.99x).

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants