Skip to content

fix(layout): normalize child validity beneath nullable structs - #11160

Draft
gatesn wants to merge 9 commits into
ngates/compress-runend-sequencesfrom
ngates/normalize-struct-child-validity
Draft

gatesn wants to merge 9 commits into
ngates/compress-runend-sequencesfrom
ngates/normalize-struct-child-validity

Conversation

@gatesn

@gatesn gatesn commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

The struct layout writer normalizes the validity of children beneath nullable structs before splitting a chunk into columns. When an ancestor struct is null at a row, nothing below it at that row is observable. The writer therefore marks inner structs valid at those rows, and replaces hidden leaf values with the type's default, also valid. Visible rows and visible nulls are untouched, and the outermost validity still records which rows are null, so every value read through the struct is identical. On disk, each level no longer repeats its ancestors' null bitmap, and hidden leaf rows no longer carry arbitrary physical bytes into the compressor. Arrow-sourced nulls, for example, can hold any view or offset. In wide, sparse, nested data, that redundant and noisy content dominates a column.

Defaults are substituted for Bool, Primitive, Decimal, Utf8 and Binary leaves. List and FixedSizeList leaves keep their own nulls: selecting into a list canonicalizes it to a ListView, which compresses and decodes far worse than the incoming list. Union and variant leaves have no cheap default and are also left alone. The leaf rewrite executes the selection with RecursiveCanonical, so the writers receive a materialized validity bitmap rather than a lazy zip.

For review: check that the reader always reapplies ancestor validity when a nested field is projected or filtered on its own. A child column read without its ancestors would show defaults where the input had nulls; the struct_::reader nested-validity tests cover projection, filtering and is_null at each level. Also review normalize_struct_children in layouts/struct_/normalize.rs: it recurses on visible & validity and widens each struct's validity to validity | !visible.

Compression bench on the sparse-wide datasets, together with #11159 (this PR's base), against develop. Hot times; every other dataset is byte-identical:

sparse-wide dataset Vortex file size compress decompress
nested-child-nulls 1.69 → 1.39 MB (−17.8%) +0.7% −16.5%
lists-runs 3.95 → 3.37 MB (−14.7%) +0.0% −12.1%
lists-shuffled 6.44 → 5.66 MB (−12.1%) +92.2% −11.2%
nested-runs 1.30 → 1.27 MB (−2.0%) +2.7% −21.1%
flat-runs unchanged −1.6% +0.4%

#11159 alone accounts for −12.0% on lists-runs; the rest of the size reduction comes from this PR. In CI, lists-shuffled compresses in 562–570 ms across three runs, against 296 ms on develop. That matches lists-runs, which takes 678 ms on develop. Nothing in that dataset's leaves is rewritten; only struct validity changes. The slowdown does not reproduce on an Apple M-series machine with the bench's CI settings (FLAT_LAYOUT_INLINE_ARRAY_NODE=1), even though the files written are byte-identical to CI's. There, single-threaded compression of the list datasets costs 2–7% more, normalization accounts for 1–2% of CPU, and 16-thread timings are unchanged. The CI slowdown is specific to the benchmark hardware and remains to be profiled there.

Stacked on #11159.

🤖 Generated with Claude Code

Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn changed the base branch from develop to ngates/compress-runend-sequences November 20, 2026 17:13
@gatesn gatesn added the action/bench-compress Run only the compression benchmark on this PR label Nov 20, 2026
@github-actions github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Nov 20, 2026
Signed-off-by: Nicholas Gates <nick@nickgates.com>
I, Nicholas Gates <nick@nickgates.com>, hereby add my Signed-off-by to this commit: 239f6c7
I, Nicholas Gates <nick@nickgates.com>, hereby add my Signed-off-by to this commit: fe46c27

Signed-off-by: Nicholas Gates <nick@nickgates.com>
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 0d7188e compress-bench 1 Explore Profiling Data
Previous Runs (3)
Status Commit Job Attempt Link
🟢 Done 3b9b0b8 compress-bench 1 Explore Profiling Data
🟢 Done 608664f compress-bench 1 Explore Profiling Data
🟢 Done fe46c27 compress-bench 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Benchmarks: Compression 📖

Commits: PR 0d7188e5 vs base a0e86198


vortex / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 455017858 / 451183124 / +0.8% 455063467 / 451363193 / +0.8% 1.00 / 1.00 / +0.0%
compress time/bimbo 2380030185 / 2333904726 / +2.0% 2357532890 / 2326781638 / +1.3% 1.01 / 1.00 / +0.6%
compress time/cmsprovider 1053247284 / 1046352758 / +0.7% 1037988148 / 1033067553 / +0.5% 1.01 / 1.01 / +0.2%
compress time/euro2016 179428240 / 177976820 / +0.8% 179070664 / 177848006 / +0.7% 1.00 / 1.00 / +0.1%
compress time/food 209196794 / 207056266 / +1.0% 210340032 / 208051568 / +1.1% 0.99 / 1.00 / -0.1%
compress time/hashtags 369261682 / 368625558 / +0.2% 358663773 / 362462710 / -1.0% 1.03 / 1.02 / +1.2%
compress time/sparse-wide-flat-runs 89256806 / 91702580 / -2.7% 93326080 / 90830808 / +2.7% 0.96 / 1.01 / -5.3%
compress time/sparse-wide-lists-runs 671733355 / 677954217 / -0.9% 667090064 / 686865605 / -2.9% 1.01 / 0.99 / +2.0%
compress time/sparse-wide-lists-shuffled 303012828 / 296469154 / +2.2% 301677650 / 291289522 / +3.6% 1.00 / 1.02 / -1.3%
compress time/sparse-wide-nested-child-nulls 637843770 / 629801624 / +1.3% 629286303 / 647330143 / -2.8% 1.01 / 0.97 / +4.2%
compress time/sparse-wide-nested-runs 616652267 / 616956029 / -0.0% 624876744 / 618541120 / +1.0% 0.99 / 1.00 / -1.1%
compress time/statpopgen 487404212 / 492446258 / -1.0% 505704219 / 491872220 / +2.8% 0.96 / 1.00 / -3.7%
compress time/taxi 208685620 / 206179658 / +1.2% 209579003 / 207695789 / +0.9% 1.00 / 0.99 / +0.3%
compress time/tpc-h l_comment canonical 108278506 / 107957618 / +0.3% 109633583 / 109240370 / +0.4% 0.99 / 0.99 / -0.1%
compress time/tpc-h l_comment chunked 108030528 / 107967360 / +0.1% 109314300 / 109647394 / -0.3% 0.99 / 0.98 / +0.4%
compress time/tpc-h sf1 lineitem 400984732 / 396066838 / +1.2% 397550145 / 396762827 / +0.2% 1.01 / 1.00 / +1.0%
compress time/tpc-h sf1 orders 87916406 / 87336481 / +0.7% 88503993 / 87528582 / +1.1% 0.99 / 1.00 / -0.4%
compress time/tpc-h sf1 partsupp 59045240 / 58822059 / +0.4% 59687319 / 59319582 / +0.6% 0.99 / 0.99 / -0.2%
compress time/wide table cols=100 chunks=1 rows=1000 5467342 / 5589374 / -2.2% 7917095 / 7105993 / +11.4% 🔴 0.69 / 0.79 / -12.2%
compress time/wide table cols=100 chunks=50 rows=1000 5635228 / 5658000 / -0.4% 8282898 / 7447890 / +11.2% 🔴 0.68 / 0.76 / -10.4%
compress time/wide table cols=1000 chunks=1 rows=1000 55746452 / 55248425 / +0.9% 57016098 / 59758853 / -4.6% 0.98 / 0.92 / +5.8%
compress time/wide table cols=1000 chunks=50 rows=1000 55769368 / 55469542 / +0.5% 57473226 / 56585778 / +1.6% 0.97 / 0.98 / -1.0%
compress time/wide table cols=10000 chunks=1 rows=1000 629227998 / 632522014 / -0.5% 626329981 / 625075396 / +0.2% 1.00 / 1.01 / -0.7%
compress time/wide table cols=10000 chunks=50 rows=1000 639789348 / 632294036 / +1.2% 622790673 / 624951888 / -0.3% 1.03 / 1.01 / +1.5%
decompress time/arade 14968108 / 14963379 / +0.0% 17162227 / 15703632 / +9.3% 0.87 / 0.95 / -8.5%
decompress time/bimbo 69142833 / 68625536 / +0.8% 75246491 / 74933035 / +0.4% 0.92 / 0.92 / +0.3%
decompress time/cmsprovider 39567258 / 39761912 / -0.5% 41691657 / 40194812 / +3.7% 0.95 / 0.99 / -4.1%
decompress time/euro2016 12580012 / 12534916 / +0.4% 13995593 / 13326348 / +5.0% 0.90 / 0.94 / -4.4%
decompress time/food 4226736 / 4019294 / +5.2% 5506898 / 4565280 / +20.6% 🔴 0.77 / 0.88 / -12.8%
decompress time/hashtags 57761940 / 57399539 / +0.6% 58032651 / 56627641 / +2.5% 1.00 / 1.01 / -1.8%
decompress time/sparse-wide-flat-runs 14214596 / 13943213 / +1.9% 13351100 / 13957497 / -4.3% 1.06 / 1.00 / +6.6%
decompress time/sparse-wide-lists-runs 16016452 / 16551077 / -3.2% 17618647 / 16922893 / +4.1% 0.91 / 0.98 / -7.1%
decompress time/sparse-wide-lists-shuffled 26715540 / 26612868 / +0.4% 28424769 / 28273138 / +0.5% 0.94 / 0.94 / -0.1%
decompress time/sparse-wide-nested-child-nulls 16279056 / 17740109 / -8.2% 19020824 / 17582474 / +8.2% 0.86 / 1.01 / -15.2%
decompress time/sparse-wide-nested-runs 15698248 / 16526684 / -5.0% 22601190 / 15589785 / +45.0% 🔴 0.69 / 1.06 / -34.5%
decompress time/statpopgen 698903734 / 714170360 / -2.1% 716032325 / 713852106 / +0.3% 0.98 / 1.00 / -2.4%
decompress time/taxi 10655685 / 10903533 / -2.3% 12465188 / 11683556 / +6.7% 0.85 / 0.93 / -8.4%
decompress time/tpc-h l_comment canonical 5147426 / 5185414 / -0.7% 6263194 / 5674115 / +10.4% 🔴 0.82 / 0.91 / -10.1%
decompress time/tpc-h l_comment chunked 5144558 / 5159250 / -0.3% 5881679 / 5691830 / +3.3% 0.87 / 0.91 / -3.5%
decompress time/tpc-h sf1 lineitem 15728665 / 15610874 / +0.8% 16532380 / 16634371 / -0.6% 0.95 / 0.94 / +1.4%
decompress time/tpc-h sf1 orders 4468220 / 4146764 / +7.8% 4735270 / 5070920 / -6.6% 0.94 / 0.82 / +15.4%
decompress time/tpc-h sf1 partsupp 3280938 / 3309534 / -0.9% 4378499 / 3863813 / +13.3% 🔴 0.75 / 0.86 / -12.5%
decompress time/wide table cols=100 chunks=1 rows=1000 1528825 / 1562372 / -2.1% 2177277 / 1689244 / +28.9% 🔴 0.70 / 0.92 / -24.1%
decompress time/wide table cols=100 chunks=50 rows=1000 1540063 / 1581660 / -2.6% 2146796 / 1679370 / +27.8% 🔴 0.72 / 0.94 / -23.8%
decompress time/wide table cols=1000 chunks=1 rows=1000 14008266 / 13745262 / +1.9% 15100679 / 14338020 / +5.3% 0.93 / 0.96 / -3.2%
decompress time/wide table cols=1000 chunks=50 rows=1000 13940512 / 13935021 / +0.0% 14878011 / 14435443 / +3.1% 0.94 / 0.97 / -2.9%
decompress time/wide table cols=10000 chunks=1 rows=1000 145597946 / 143078716 / +1.8% 150047345 / 145481429 / +3.1% 0.97 / 0.98 / -1.3%
decompress time/wide table cols=10000 chunks=50 rows=1000 146941616 / 143281848 / +2.6% 149213361 / 144600834 / +3.2% 0.98 / 0.99 / -0.6%
vortex / vortex-file-compressed / bytes (0.992x ➖, 2↑ 0↓)
name bytes (PR / base / %diff)
vortex-file-compressed size/arade 141323316 / 141323316 / +0.0%
vortex-file-compressed size/bimbo 410263708 / 410263708 / +0.0%
vortex-file-compressed size/cmsprovider 413116844 / 413116844 / +0.0%
vortex-file-compressed size/euro2016 152436356 / 152436356 / +0.0%
vortex-file-compressed size/food 37247744 / 37247744 / +0.0%
vortex-file-compressed size/hashtags 169636564 / 169636564 / +0.0%
vortex-file-compressed size/sparse-wide-flat-runs 1552504 / 1552504 / +0.0%
vortex-file-compressed size/sparse-wide-lists-runs 3473956 / 3948028 / -12.0% 🟢
vortex-file-compressed size/sparse-wide-lists-shuffled 6436588 / 6436588 / +0.0%
vortex-file-compressed size/sparse-wide-nested-child-nulls 1495772 / 1687476 / -11.4% 🟢
vortex-file-compressed size/sparse-wide-nested-runs 1380404 / 1298532 / +6.3%
vortex-file-compressed size/statpopgen 232298372 / 232298372 / +0.0%
vortex-file-compressed size/taxi 48083012 / 48083012 / +0.0%
vortex-file-compressed size/tpc-h l_comment canonical 64987292 / 64987292 / +0.0%
vortex-file-compressed size/tpc-h l_comment chunked 64987292 / 64987292 / +0.0%
vortex-file-compressed size/tpc-h sf1 lineitem 154809600 / 154809600 / +0.0%
vortex-file-compressed size/tpc-h sf1 orders 33372900 / 33372900 / +0.0%
vortex-file-compressed size/tpc-h sf1 partsupp 24526904 / 24526904 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000 934968 / 934968 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000 934968 / 934968 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000 9333768 / 9333768 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000 9333768 / 9333768 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000 93357768 / 93357768 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000 93357768 / 93357768 / +0.0%
vortex / parquet / ns (0.996x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 1158339368 / 1159851524 / -0.1% 1156291816 / 1158311510 / -0.2% 1.00 / 1.00 / +0.0%
compress time/bimbo 2121129812 / 2133056944 / -0.6% 2126072897 / 2136148566 / -0.5% 1.00 / 1.00 / -0.1%
compress time/cmsprovider 1664816770 / 1667948089 / -0.2% 1669582747 / 1679133907 / -0.6% 1.00 / 0.99 / +0.4%
compress time/euro2016 994107540 / 998128944 / -0.4% 995798385 / 1005511543 / -1.0% 1.00 / 0.99 / +0.6%
compress time/food 405735584 / 406076426 / -0.1% 407581312 / 408110340 / -0.1% 1.00 / 1.00 / +0.0%
compress time/hashtags 537533746 / 536618941 / +0.2% 545380114 / 546287947 / -0.2% 0.99 / 0.98 / +0.3%
compress time/sparse-wide-flat-runs 41251122 / 41944672 / -1.7% 42312433 / 41984931 / +0.8% 0.97 / 1.00 / -2.4%
compress time/sparse-wide-lists-runs 81329826 / 81515742 / -0.2% 81915721 / 82384507 / -0.6% 0.99 / 0.99 / +0.3%
compress time/sparse-wide-lists-shuffled 154839880 / 155754334 / -0.6% 155130041 / 155999796 / -0.6% 1.00 / 1.00 / -0.0%
compress time/sparse-wide-nested-child-nulls 54073306 / 54601914 / -1.0% 54285772 / 55157956 / -1.6% 1.00 / 0.99 / +0.6%
compress time/sparse-wide-nested-runs 55636982 / 55310942 / +0.6% 56171595 / 55492519 / +1.2% 0.99 / 1.00 / -0.6%
compress time/statpopgen 1029763420 / 1024981765 / +0.5% 1035886785 / 1029023341 / +0.7% 0.99 / 1.00 / -0.2%
compress time/taxi 239270300 / 239396698 / -0.1% 240042015 / 241349101 / -0.5% 1.00 / 0.99 / +0.5%
compress time/tpc-h l_comment canonical 721155762 / 720041870 / +0.2% 723856337 / 723018526 / +0.1% 1.00 / 1.00 / +0.0%
compress time/tpc-h l_comment chunked 721779483 / 721483056 / +0.0% 723596022 / 723284033 / +0.0% 1.00 / 1.00 / -0.0%
compress time/tpc-h sf1 lineitem 1202889444 / 1201547807 / +0.1% 1216166660 / 1209394080 / +0.6% 0.99 / 0.99 / -0.4%
compress time/tpc-h sf1 orders 337704956 / 339165898 / -0.4% 350127947 / 355015699 / -1.4% 0.96 / 0.96 / +1.0%
compress time/tpc-h sf1 partsupp 305172214 / 306386224 / -0.4% 309351187 / 307383831 / +0.6% 0.99 / 1.00 / -1.0%
compress time/wide table cols=100 chunks=1 rows=1000 3854024 / 3748931 / +2.8% 4355513 / 4398059 / -1.0% 0.88 / 0.85 / +3.8%
compress time/wide table cols=100 chunks=50 rows=1000 3629070 / 3743644 / -3.1% 4329551 / 4546978 / -4.8% 0.84 / 0.82 / +1.8%
compress time/wide table cols=1000 chunks=1 rows=1000 37225517 / 37707725 / -1.3% 38915579 / 39201274 / -0.7% 0.96 / 0.96 / -0.6%
compress time/wide table cols=1000 chunks=50 rows=1000 37326578 / 37203558 / +0.3% 38227901 / 39017799 / -2.0% 0.98 / 0.95 / +2.4%
compress time/wide table cols=10000 chunks=1 rows=1000 376392176 / 377630973 / -0.3% 384926400 / 389773560 / -1.2% 0.98 / 0.97 / +0.9%
compress time/wide table cols=10000 chunks=50 rows=1000 374367696 / 378613433 / -1.1% 380488802 / 384293384 / -1.0% 0.98 / 0.99 / -0.1%
decompress time/arade 54523584 / 54601782 / -0.1% 54657079 / 54859370 / -0.4% 1.00 / 1.00 / +0.2%
decompress time/bimbo 92259689 / 92318156 / -0.1% 92611046 / 92679138 / -0.1% 1.00 / 1.00 / +0.0%
decompress time/cmsprovider 103135478 / 103251782 / -0.1% 103074808 / 103175716 / -0.1% 1.00 / 1.00 / -0.0%
decompress time/euro2016 159233104 / 159426388 / -0.1% 159696989 / 159723065 / -0.0% 1.00 / 1.00 / -0.1%
decompress time/food 26569976 / 26602534 / -0.1% 26650772 / 26742726 / -0.3% 1.00 / 0.99 / +0.2%
decompress time/hashtags 445646815 / 445133732 / +0.1% 445284210 / 445350725 / -0.0% 1.00 / 1.00 / +0.1%
decompress time/sparse-wide-flat-runs 15005796 / 15276262 / -1.8% 15437371 / 15883138 / -2.8% 0.97 / 0.96 / +1.1%
decompress time/sparse-wide-lists-runs 277563542 / 277390748 / +0.1% 277391346 / 278014265 / -0.2% 1.00 / 1.00 / +0.3%
decompress time/sparse-wide-lists-shuffled 318005710 / 319073929 / -0.3% 318462139 / 319901475 / -0.4% 1.00 / 1.00 / +0.1%
decompress time/sparse-wide-nested-child-nulls 52952744 / 52809850 / +0.3% 53160218 / 53137390 / +0.0% 1.00 / 0.99 / +0.2%
decompress time/sparse-wide-nested-runs 52854704 / 52843989 / +0.0% 52969985 / 53169078 / -0.4% 1.00 / 0.99 / +0.4%
decompress time/statpopgen 2918304924 / 2928301498 / -0.3% 2967117341 / 2992571106 / -0.9% 0.98 / 0.98 / +0.5%
decompress time/taxi 60493422 / 60795858 / -0.5% 60936962 / 61049394 / -0.2% 0.99 / 1.00 / -0.3%
decompress time/tpc-h l_comment canonical 35946002 / 35994949 / -0.1% 36085316 / 36091820 / -0.0% 1.00 / 1.00 / -0.1%
decompress time/tpc-h l_comment chunked 35947316 / 35999986 / -0.1% 36145513 / 36126936 / +0.1% 0.99 / 1.00 / -0.2%
decompress time/tpc-h sf1 lineitem 82319810 / 82255822 / +0.1% 82467824 / 82329605 / +0.2% 1.00 / 1.00 / -0.1%
decompress time/tpc-h sf1 orders 87957632 / 88171542 / -0.2% 88465922 / 88469438 / -0.0% 0.99 / 1.00 / -0.2%
decompress time/tpc-h sf1 partsupp 110923052 / 111201218 / -0.3% 111266032 / 111755942 / -0.4% 1.00 / 1.00 / +0.2%
decompress time/wide table cols=100 chunks=1 rows=1000 1927270 / 1956334 / -1.5% 2121925 / 2083166 / +1.9% 0.91 / 0.94 / -3.3%
decompress time/wide table cols=100 chunks=50 rows=1000 1944444 / 1950982 / -0.3% 2122834 / 2112020 / +0.5% 0.92 / 0.92 / -0.8%
decompress time/wide table cols=1000 chunks=1 rows=1000 22594552 / 22819352 / -1.0% 23035237 / 23784360 / -3.1% 0.98 / 0.96 / +2.2%
decompress time/wide table cols=1000 chunks=50 rows=1000 22323211 / 22613362 / -1.3% 22795770 / 23761161 / -4.1% 0.98 / 0.95 / +2.9%
decompress time/wide table cols=10000 chunks=1 rows=1000 231548230 / 233845626 / -1.0% 233317836 / 236270815 / -1.2% 0.99 / 0.99 / +0.3%
decompress time/wide table cols=10000 chunks=50 rows=1000 230424884 / 234363140 / -1.7% 233260203 / 236747868 / -1.5% 0.99 / 0.99 / -0.2%
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
parquet size/arade 210020851 / 210020851 / +0.0%
parquet size/bimbo 376187078 / 376187078 / +0.0%
parquet size/cmsprovider 364268159 / 364268159 / +0.0%
parquet size/euro2016 111035158 / 111035158 / +0.0%
parquet size/food 34903783 / 34903783 / +0.0%
parquet size/hashtags 126189203 / 126189203 / +0.0%
parquet size/sparse-wide-flat-runs 1388944 / 1388944 / +0.0%
parquet size/sparse-wide-lists-runs 4327813 / 4327813 / +0.0%
parquet size/sparse-wide-lists-shuffled 5710569 / 5710569 / +0.0%
parquet size/sparse-wide-nested-child-nulls 1266042 / 1266042 / +0.0%
parquet size/sparse-wide-nested-runs 1388944 / 1388944 / +0.0%
parquet size/statpopgen 81284006 / 81284006 / +0.0%
parquet size/taxi 52142534 / 52142534 / +0.0%
parquet size/tpc-h l_comment canonical 52204116 / 52204116 / +0.0%
parquet size/tpc-h l_comment chunked 52204116 / 52204116 / +0.0%
parquet size/tpc-h sf1 lineitem 153403836 / 153403836 / +0.0%
parquet size/tpc-h sf1 orders 37029542 / 37029542 / +0.0%
parquet size/tpc-h sf1 partsupp 29200612 / 29200612 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000 93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000 93240004 / 93240004 / +0.0%
vortex / arrow-ipc / ns (0.992x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 95010200 / 95865668 / -0.9% 118535636 / 122415926 / -3.2% 0.80 / 0.78 / +2.4%
compress time/bimbo 593405764 / 603397805 / -1.7% 655543992 / 668158410 / -1.9% 0.91 / 0.90 / +0.2%
compress time/cmsprovider 373688094 / 380564888 / -1.8% 435664789 / 445103703 / -2.1% 0.86 / 0.86 / +0.3%
compress time/euro2016 38072057 / 38319792 / -0.6% 51577237 / 52782696 / -2.3% 0.74 / 0.73 / +1.7%
compress time/food 39943688 / 40343196 / -1.0% 53185806 / 54048118 / -1.6% 0.75 / 0.75 / +0.6%
compress time/hashtags 77471313 / 78276669 / -1.0% 105506732 / 106901128 / -1.3% 0.73 / 0.73 / +0.3%
compress time/sparse-wide-flat-runs 27772818 / 27922224 / -0.5% 36615271 / 37253526 / -1.7% 0.76 / 0.75 / +1.2%
compress time/sparse-wide-lists-runs 44873622 / 45075541 / -0.4% 51337126 / 51993300 / -1.3% 0.87 / 0.87 / +0.8%
compress time/sparse-wide-lists-shuffled 46504592 / 46588675 / -0.2% 49816980 / 50386218 / -1.1% 0.93 / 0.92 / +1.0%
compress time/sparse-wide-nested-child-nulls 40397376 / 40864326 / -1.1% 49991426 / 50936751 / -1.9% 0.81 / 0.80 / +0.7%
compress time/sparse-wide-nested-runs 40442160 / 40676459 / -0.6% 50203928 / 50782898 / -1.1% 0.81 / 0.80 / +0.6%
compress time/statpopgen 216014372 / 216330328 / -0.1% 262017564 / 262890095 / -0.3% 0.82 / 0.82 / +0.2%
compress time/taxi 58315453 / 59255102 / -1.6% 74392044 / 76397935 / -2.6% 0.78 / 0.78 / +1.1%
compress time/tpc-h l_comment canonical 22797266 / 22571036 / +1.0% 30648391 / 30895072 / -0.8% 0.74 / 0.73 / +1.8%
compress time/tpc-h l_comment chunked 22740834 / 22513906 / +1.0% 30359083 / 30838197 / -1.6% 0.75 / 0.73 / +2.6%
compress time/tpc-h sf1 lineitem 127377578 / 128575520 / -0.9% 162463070 / 165030849 / -1.6% 0.78 / 0.78 / +0.6%
compress time/tpc-h sf1 orders 24520168 / 24682734 / -0.7% 34191679 / 34551466 / -1.0% 0.72 / 0.71 / +0.4%
compress time/tpc-h sf1 partsupp 13101543 / 13158730 / -0.4% 19111495 / 19405427 / -1.5% 0.69 / 0.68 / +1.1%
compress time/wide table cols=100 chunks=1 rows=1000 192812.5 / 206229 / -6.5% 438789 / 652732 / -32.8% 🟢 0.44 / 0.32 / +39.1%
compress time/wide table cols=100 chunks=50 rows=1000 194570.5 / 199623.5 / -2.5% 375171 / 622612 / -39.7% 🟢 0.52 / 0.32 / +61.8%
compress time/wide table cols=1000 chunks=1 rows=1000 2326767 / 2341174 / -0.6% 2812936 / 3186366 / -11.7% 🟢 0.83 / 0.73 / +12.6%
compress time/wide table cols=1000 chunks=50 rows=1000 2289710 / 2320939 / -1.3% 2798599 / 3082409 / -9.2% 0.82 / 0.75 / +8.7%
compress time/wide table cols=10000 chunks=1 rows=1000 24903103 / 24878252 / +0.1% 30886816 / 31350770 / -1.5% 0.81 / 0.79 / +1.6%
compress time/wide table cols=10000 chunks=50 rows=1000 24968218 / 24957414 / +0.0% 31148110 / 31501179 / -1.1% 0.80 / 0.79 / +1.2%
decompress time/arade 209679462 / 210239864 / -0.3% 209811913 / 210426535 / -0.3% 1.00 / 1.00 / +0.0%
decompress time/bimbo 398374320 / 424705086 / -6.2% 407205173 / 424732118 / -4.1% 0.98 / 1.00 / -2.2%
decompress time/cmsprovider 694066626 / 674213448 / +2.9% 694624997 / 674817862 / +2.9% 1.00 / 1.00 / +0.0%
decompress time/euro2016 83506064 / 83465004 / +0.0% 83788836 / 83689290 / +0.1% 1.00 / 1.00 / -0.1%
decompress time/food 65317753 / 67242871 / -2.9% 65442043 / 67343455 / -2.8% 1.00 / 1.00 / -0.0%
decompress time/hashtags 159453902 / 160347404 / -0.6% 159796873 / 160758461 / -0.6% 1.00 / 1.00 / +0.0%
decompress time/sparse-wide-flat-runs 18702994 / 18747056 / -0.2% 20172512 / 19847949 / +1.6% 0.93 / 0.94 / -1.8%
decompress time/sparse-wide-lists-runs 81301526 / 81718661 / -0.5% 81377452 / 81853158 / -0.6% 1.00 / 1.00 / +0.1%
decompress time/sparse-wide-lists-shuffled 81651390 / 81934875 / -0.3% 81773368 / 82021176 / -0.3% 1.00 / 1.00 / -0.0%
decompress time/sparse-wide-nested-child-nulls 26280255 / 26583464 / -1.1% 26878593 / 27717418 / -3.0% 0.98 / 0.96 / +1.9%
decompress time/sparse-wide-nested-runs 26629068 / 26382980 / +0.9% 28109455 / 26868869 / +4.6% 0.95 / 0.98 / -3.5%
decompress time/statpopgen 462798348 / 469823662 / -1.5% 462068514 / 470310750 / -1.8% 1.00 / 1.00 / +0.3%
decompress time/taxi 50719688 / 50900952 / -0.4% 50946128 / 51131487 / -0.4% 1.00 / 1.00 / +0.0%
decompress time/tpc-h l_comment canonical 39668875 / 40921581 / -3.1% 39916154 / 41228932 / -3.2% 0.99 / 0.99 / +0.1%
decompress time/tpc-h l_comment chunked 39762418 / 40857360 / -2.7% 39960644 / 41224683 / -3.1% 1.00 / 0.99 / +0.4%
decompress time/tpc-h sf1 lineitem 191122432 / 191759513 / -0.3% 191344416 / 191823063 / -0.2% 1.00 / 1.00 / -0.1%
decompress time/tpc-h sf1 orders 39362050 / 39546924 / -0.5% 39471359 / 39636637 / -0.4% 1.00 / 1.00 / -0.1%
decompress time/tpc-h sf1 partsupp 13776000 / 13864499 / -0.6% 13917393 / 14039661 / -0.9% 0.99 / 0.99 / +0.2%
decompress time/wide table cols=100 chunks=1 rows=1000 313740 / 317044 / -1.0% 338791 / 338859 / -0.0% 0.93 / 0.94 / -1.0%
decompress time/wide table cols=100 chunks=50 rows=1000 317963 / 316698 / +0.4% 336461 / 341353 / -1.4% 0.95 / 0.93 / +1.9%
decompress time/wide table cols=1000 chunks=1 rows=1000 3708144 / 3528793 / +5.1% 3892470 / 3784174 / +2.9% 0.95 / 0.93 / +2.2%
decompress time/wide table cols=1000 chunks=50 rows=1000 3654322 / 3783188 / -3.4% 3739858 / 3776298 / -1.0% 0.98 / 1.00 / -2.5%
decompress time/wide table cols=10000 chunks=1 rows=1000 35392132 / 35619092 / -0.6% 35677233 / 36139101 / -1.3% 0.99 / 0.99 / +0.6%
decompress time/wide table cols=10000 chunks=50 rows=1000 36209006 / 36336174 / -0.3% 36410353 / 36983653 / -1.6% 0.99 / 0.98 / +1.2%
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
arrow-ipc size/arade 680639722 / 680639722 / +0.0%
arrow-ipc size/bimbo 3168319202 / 3168319202 / +0.0%
arrow-ipc size/cmsprovider 2394412322 / 2394412322 / +0.0%
arrow-ipc size/euro2016 359300042 / 359300042 / +0.0%
arrow-ipc size/food 252069442 / 252069442 / +0.0%
arrow-ipc size/hashtags 700364562 / 700364562 / +0.0%
arrow-ipc size/sparse-wide-flat-runs 245220370 / 245220370 / +0.0%
arrow-ipc size/sparse-wide-lists-runs 142046306 / 142046306 / +0.0%
arrow-ipc size/sparse-wide-lists-shuffled 142870882 / 142870882 / +0.0%
arrow-ipc size/sparse-wide-nested-child-nulls 251527506 / 251527506 / +0.0%
arrow-ipc size/sparse-wide-nested-runs 251527122 / 251527122 / +0.0%
arrow-ipc size/statpopgen 1890451826 / 1890451826 / +0.0%
arrow-ipc size/taxi 474481242 / 474481242 / +0.0%
arrow-ipc size/tpc-h l_comment canonical 184829074 / 184829074 / +0.0%
arrow-ipc size/tpc-h l_comment chunked 184829074 / 184829074 / +0.0%
arrow-ipc size/tpc-h sf1 lineitem 1021072290 / 1021072290 / +0.0%
arrow-ipc size/tpc-h sf1 orders 194812026 / 194812026 / +0.0%
arrow-ipc size/tpc-h sf1 partsupp 131286562 / 131286562 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000 1231690 / 1231690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000 1231690 / 1231690 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000 12312506 / 12312506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000 12312506 / 12312506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000 123192506 / 123192506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000 123192506 / 123192506 / +0.0%

@robert3005

Copy link
Copy Markdown
Contributor

You have pretty bad regressions in the compress benchmarks :/

gatesn and others added 2 commits November 20, 2026 12:58
Selecting defaults into a list canonicalizes it to a ListView, which made the
sparse-wide list datasets larger and several times slower to compress and
decode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn added the action/bench-compress Run only the compression benchmark on this PR label Nov 20, 2026
@github-actions github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Nov 20, 2026
@gatesn gatesn added the changelog/performance A performance improvement label Nov 20, 2026
Zip leaves its validity as a lazy zip array, which the flat writer cannot
serialize.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn added the action/bench-compress Run only the compression benchmark on this PR label Nov 20, 2026
@github-actions github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Nov 20, 2026
@gatesn gatesn added the action/bench-compress Run only the compression benchmark on this PR label Nov 20, 2026
@github-actions github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Nov 20, 2026
Temporary, to measure the leaf rewrite apart from collapsing struct validity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn added the action/bench-compress Run only the compression benchmark on this PR label Nov 20, 2026
@github-actions github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants