Skip to content

feat(compressor): encode run-end sequences with terminal exceptions - #11159

Draft
gatesn wants to merge 6 commits into
developfrom
ngates/compress-runend-sequences
Draft

gatesn wants to merge 6 commits into
developfrom
ngates/compress-runend-sequences

Conversation

@gatesn

@gatesn gatesn commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Adds a BtrBlocks integer scheme, vortex.int.runend_sequence, for the run-ends child of a run-end array. It encodes the ends as Chunked[Sequence, Constant(1)]: an arithmetic prefix followed by a single terminal value. This is the shape run-end encoding produces for the offsets of a sparse list column whose present rows are clustered. Rows before the cluster form one long run, each present row is its own run, and the rows after it form one long final run. The ends are therefore start, start+1, …, start+k, len. Everything but the last value is a sequence, so the whole child costs a few dozen bytes, whatever the row count. A plain Sequence cannot represent it because of the terminal value, while bit-packing or delta pays per run. The run values (0, 3, 6, …) are already caught by the existing sequence scheme.

The scheme is gated to stay cheap on every other integer column. It only runs on the ends child of a run-end cascade, never on samples, and only on non-null arrays of at least four values. Its estimate is deferred and exact: it encodes the prefix with sequence_encode and is skipped up front when the best scheme so far already beats the ratio this one could reach. It is excluded from the CUDA compression mode, like the other schemes without a GPU decode path.

For review:

  • the cascade-history gate (RunEndScheme.id(), 1), which must select the ends child and not the values;
  • the upper bound (nbytes / 24) used to skip the attempt;
  • the CUDA exclusion in builder.rs.

The new test compresses clamped, list-offset-shaped data and checks that the ends become Chunked.

Compression bench on the sparse-wide datasets, against develop. Hot times; every other dataset is byte-identical:

sparse-wide dataset Vortex file size compress decompress
lists-runs 3.95 → 3.47 MB (−12.0%) −3.2% −2.9%
lists-shuffled unchanged +2.5% +1.1%
flat-runs unchanged +1.5% −6.9%
nested-runs unchanged −2.7% −7.5%
nested-child-nulls unchanged −4.4% −12.5%

The timing differences on datasets whose files are unchanged are run-to-run noise. Shuffled lists don't benefit, because their present rows aren't contiguous, so their offsets don't form a single arithmetic run. #11160 builds on this for nested and shuffled data.

🤖 Generated with Claude Code

Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn added the action/bench-compress Run only the compression benchmark on this PR label Nov 20, 2026
@github-actions github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Nov 20, 2026
I, Nicholas Gates <nick@nickgates.com>, hereby add my Signed-off-by to this commit: ee8a9b3
I, Nicholas Gates <nick@nickgates.com>, hereby add my Signed-off-by to this commit: f97753e
I, Nicholas Gates <nick@nickgates.com>, hereby add my Signed-off-by to this commit: f7eb9e2

Signed-off-by: Nicholas Gates <nick@nickgates.com>
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done f7eb9e2 compress-bench 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

Copy link
Copy Markdown
Contributor

Benchmarks: Compression 📖

Commits: PR f7eb9e25 vs base a0e86198


vortex / vortex-file-compressed / ns (0.994x ➖, 1↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 451622758 / 451183124 / +0.1% 451361219 / 451363193 / -0.0% 1.00 / 1.00 / +0.1%
compress time/bimbo 2333936462 / 2333904726 / +0.0% 2329725980 / 2326781638 / +0.1% 1.00 / 1.00 / -0.1%
compress time/cmsprovider 1045542496 / 1046352758 / -0.1% 1030929898 / 1033067553 / -0.2% 1.01 / 1.01 / +0.1%
compress time/euro2016 179343465 / 177976820 / +0.8% 179350500 / 177848006 / +0.8% 1.00 / 1.00 / -0.1%
compress time/food 207023820 / 207056266 / -0.0% 208467483 / 208051568 / +0.2% 0.99 / 1.00 / -0.2%
compress time/hashtags 370652254 / 368625558 / +0.5% 370457911 / 362462710 / +2.2% 1.00 / 1.02 / -1.6%
compress time/sparse-wide-flat-runs 93114016 / 91702580 / +1.5% 89089959 / 90830808 / -1.9% 1.05 / 1.01 / +3.5%
compress time/sparse-wide-lists-runs 656193679 / 677954217 / -3.2% 680418074 / 686865605 / -0.9% 0.96 / 0.99 / -2.3%
compress time/sparse-wide-lists-shuffled 303918367 / 296469154 / +2.5% 318017825 / 291289522 / +9.2% 0.96 / 1.02 / -6.1%
compress time/sparse-wide-nested-child-nulls 602349854 / 629801624 / -4.4% 630633781 / 647330143 / -2.6% 0.96 / 0.97 / -1.8%
compress time/sparse-wide-nested-runs 600531244 / 616956029 / -2.7% 631943121 / 618541120 / +2.2% 0.95 / 1.00 / -4.7%
compress time/statpopgen 489216768 / 492446258 / -0.7% 533854155 / 491872220 / +8.5% 0.92 / 1.00 / -8.5%
compress time/taxi 205628990 / 206179658 / -0.3% 205988299 / 207695789 / -0.8% 1.00 / 0.99 / +0.6%
compress time/tpc-h l_comment canonical 108571768 / 107957618 / +0.6% 109607582 / 109240370 / +0.3% 0.99 / 0.99 / +0.2%
compress time/tpc-h l_comment chunked 107922778 / 107967360 / -0.0% 109113105 / 109647394 / -0.5% 0.99 / 0.98 / +0.4%
compress time/tpc-h sf1 lineitem 396742028 / 396066838 / +0.2% 393967486 / 396762827 / -0.7% 1.01 / 1.00 / +0.9%
compress time/tpc-h sf1 orders 87396934 / 87336481 / +0.1% 87706565 / 87528582 / +0.2% 1.00 / 1.00 / -0.1%
compress time/tpc-h sf1 partsupp 58926062 / 58822059 / +0.2% 59876466 / 59319582 / +0.9% 0.98 / 0.99 / -0.8%
compress time/wide table cols=100 chunks=1 rows=1000 5615530 / 5589374 / +0.5% 7590607 / 7105993 / +6.8% 0.74 / 0.79 / -5.9%
compress time/wide table cols=100 chunks=50 rows=1000 5705073 / 5658000 / +0.8% 7941997 / 7447890 / +6.6% 0.72 / 0.76 / -5.4%
compress time/wide table cols=1000 chunks=1 rows=1000 56389606 / 55248425 / +2.1% 56017155 / 59758853 / -6.3% 1.01 / 0.92 / +8.9%
compress time/wide table cols=1000 chunks=50 rows=1000 55287160 / 55469542 / -0.3% 58434037 / 56585778 / +3.3% 0.95 / 0.98 / -3.5%
compress time/wide table cols=10000 chunks=1 rows=1000 632807314 / 632522014 / +0.0% 625921252 / 625075396 / +0.1% 1.01 / 1.01 / -0.1%
compress time/wide table cols=10000 chunks=50 rows=1000 637379418 / 632294036 / +0.8% 620812193 / 624951888 / -0.7% 1.03 / 1.01 / +1.5%
decompress time/arade 15124404 / 14963379 / +1.1% 16620610 / 15703632 / +5.8% 0.91 / 0.95 / -4.5%
decompress time/bimbo 68347238 / 68625536 / -0.4% 74249376 / 74933035 / -0.9% 0.92 / 0.92 / +0.5%
decompress time/cmsprovider 39752447 / 39761912 / -0.0% 41806783 / 40194812 / +4.0% 0.95 / 0.99 / -3.9%
decompress time/euro2016 12749114 / 12534916 / +1.7% 14571457 / 13326348 / +9.3% 0.87 / 0.94 / -7.0%
decompress time/food 3966714 / 4019294 / -1.3% 5982800 / 4565280 / +31.1% 🔴 0.66 / 0.88 / -24.7%
decompress time/hashtags 57682679 / 57399539 / +0.5% 60517631 / 56627641 / +6.9% 0.95 / 1.01 / -6.0%
decompress time/sparse-wide-flat-runs 12986620 / 13943213 / -6.9% 12177720 / 13957497 / -12.8% 🟢 1.07 / 1.00 / +6.8%
decompress time/sparse-wide-lists-runs 16078828 / 16551077 / -2.9% 17320651 / 16922893 / +2.4% 0.93 / 0.98 / -5.1%
decompress time/sparse-wide-lists-shuffled 26893220 / 26612868 / +1.1% 28382066 / 28273138 / +0.4% 0.95 / 0.94 / +0.7%
decompress time/sparse-wide-nested-child-nulls 15525537 / 17740109 / -12.5% 🟢 16990530 / 17582474 / -3.4% 0.91 / 1.01 / -9.4%
decompress time/sparse-wide-nested-runs 15284670 / 16526684 / -7.5% 19324385 / 15589785 / +24.0% 🔴 0.79 / 1.06 / -25.4%
decompress time/statpopgen 698723467 / 714170360 / -2.2% 699816198 / 713852106 / -2.0% 1.00 / 1.00 / -0.2%
decompress time/taxi 10559807 / 10903533 / -3.2% 12317616 / 11683556 / +5.4% 0.86 / 0.93 / -8.1%
decompress time/tpc-h l_comment canonical 5132650 / 5185414 / -1.0% 6353647 / 5674115 / +12.0% 🔴 0.81 / 0.91 / -11.6%
decompress time/tpc-h l_comment chunked 5185535 / 5159250 / +0.5% 6481877 / 5691830 / +13.9% 🔴 0.80 / 0.91 / -11.7%
decompress time/tpc-h sf1 lineitem 15644410 / 15610874 / +0.2% 16901502 / 16634371 / +1.6% 0.93 / 0.94 / -1.4%
decompress time/tpc-h sf1 orders 4161376 / 4146764 / +0.4% 5806789 / 5070920 / +14.5% 🔴 0.72 / 0.82 / -12.4%
decompress time/tpc-h sf1 partsupp 3331576 / 3309534 / +0.7% 4545370 / 3863813 / +17.6% 🔴 0.73 / 0.86 / -14.4%
decompress time/wide table cols=100 chunks=1 rows=1000 1566912 / 1562372 / +0.3% 2498730 / 1689244 / +47.9% 🔴 0.63 / 0.92 / -32.2%
decompress time/wide table cols=100 chunks=50 rows=1000 1554946 / 1581660 / -1.7% 2499823 / 1679370 / +48.9% 🔴 0.62 / 0.94 / -34.0%
decompress time/wide table cols=1000 chunks=1 rows=1000 13946480 / 13745262 / +1.5% 15304094 / 14338020 / +6.7% 0.91 / 0.96 / -4.9%
decompress time/wide table cols=1000 chunks=50 rows=1000 13887655 / 13935021 / -0.3% 15349012 / 14435443 / +6.3% 0.90 / 0.97 / -6.3%
decompress time/wide table cols=10000 chunks=1 rows=1000 145723374 / 143078716 / +1.8% 148456796 / 145481429 / +2.0% 0.98 / 0.98 / -0.2%
decompress time/wide table cols=10000 chunks=50 rows=1000 145783478 / 143281848 / +1.7% 148713336 / 144600834 / +2.8% 0.98 / 0.99 / -1.1%
vortex / vortex-file-compressed / bytes (0.995x ➖, 1↑ 0↓)
name bytes (PR / base / %diff)
vortex-file-compressed size/arade 141323316 / 141323316 / +0.0%
vortex-file-compressed size/bimbo 410263708 / 410263708 / +0.0%
vortex-file-compressed size/cmsprovider 413116844 / 413116844 / +0.0%
vortex-file-compressed size/euro2016 152436356 / 152436356 / +0.0%
vortex-file-compressed size/food 37247744 / 37247744 / +0.0%
vortex-file-compressed size/hashtags 169636564 / 169636564 / +0.0%
vortex-file-compressed size/sparse-wide-flat-runs 1552504 / 1552504 / +0.0%
vortex-file-compressed size/sparse-wide-lists-runs 3473956 / 3948028 / -12.0% 🟢
vortex-file-compressed size/sparse-wide-lists-shuffled 6436588 / 6436588 / +0.0%
vortex-file-compressed size/sparse-wide-nested-child-nulls 1687476 / 1687476 / +0.0%
vortex-file-compressed size/sparse-wide-nested-runs 1298532 / 1298532 / +0.0%
vortex-file-compressed size/statpopgen 232298372 / 232298372 / +0.0%
vortex-file-compressed size/taxi 48083012 / 48083012 / +0.0%
vortex-file-compressed size/tpc-h l_comment canonical 64987292 / 64987292 / +0.0%
vortex-file-compressed size/tpc-h l_comment chunked 64987292 / 64987292 / +0.0%
vortex-file-compressed size/tpc-h sf1 lineitem 154809600 / 154809600 / +0.0%
vortex-file-compressed size/tpc-h sf1 orders 33372900 / 33372900 / +0.0%
vortex-file-compressed size/tpc-h sf1 partsupp 24526904 / 24526904 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000 934968 / 934968 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000 934968 / 934968 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000 9333768 / 9333768 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000 9333768 / 9333768 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000 93357768 / 93357768 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000 93357768 / 93357768 / +0.0%
vortex / parquet / ns (0.996x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 1151342810 / 1159851524 / -0.7% 1164531897 / 1158311510 / +0.5% 0.99 / 1.00 / -1.3%
compress time/bimbo 2121320410 / 2133056944 / -0.6% 2121229480 / 2136148566 / -0.7% 1.00 / 1.00 / +0.1%
compress time/cmsprovider 1660641364 / 1667948089 / -0.4% 1664437794 / 1679133907 / -0.9% 1.00 / 0.99 / +0.4%
compress time/euro2016 995600566 / 998128944 / -0.3% 996838471 / 1005511543 / -0.9% 1.00 / 0.99 / +0.6%
compress time/food 406069862 / 406076426 / -0.0% 405389652 / 408110340 / -0.7% 1.00 / 1.00 / +0.7%
compress time/hashtags 538967426 / 536618941 / +0.4% 538417409 / 546287947 / -1.4% 1.00 / 0.98 / +1.9%
compress time/sparse-wide-flat-runs 41370529 / 41944672 / -1.4% 42050035 / 41984931 / +0.2% 0.98 / 1.00 / -1.5%
compress time/sparse-wide-lists-runs 81136069 / 81515742 / -0.5% 81045501 / 82384507 / -1.6% 1.00 / 0.99 / +1.2%
compress time/sparse-wide-lists-shuffled 155034917 / 155754334 / -0.5% 156743452 / 155999796 / +0.5% 0.99 / 1.00 / -0.9%
compress time/sparse-wide-nested-child-nulls 54245132 / 54601914 / -0.7% 54778000 / 55157956 / -0.7% 0.99 / 0.99 / +0.0%
compress time/sparse-wide-nested-runs 55438266 / 55310942 / +0.2% 56651637 / 55492519 / +2.1% 0.98 / 1.00 / -1.8%
compress time/statpopgen 1026743406 / 1024981765 / +0.2% 1033186291 / 1029023341 / +0.4% 0.99 / 1.00 / -0.2%
compress time/taxi 239833786 / 239396698 / +0.2% 240614655 / 241349101 / -0.3% 1.00 / 0.99 / +0.5%
compress time/tpc-h l_comment canonical 719868952 / 720041870 / -0.0% 722036794 / 723018526 / -0.1% 1.00 / 1.00 / +0.1%
compress time/tpc-h l_comment chunked 721256382 / 721483056 / -0.0% 724046568 / 723284033 / +0.1% 1.00 / 1.00 / -0.1%
compress time/tpc-h sf1 lineitem 1194022744 / 1201547807 / -0.6% 1212048838 / 1209394080 / +0.2% 0.99 / 0.99 / -0.8%
compress time/tpc-h sf1 orders 339005860 / 339165898 / -0.0% 351225850 / 355015699 / -1.1% 0.97 / 0.96 / +1.0%
compress time/tpc-h sf1 partsupp 304309106 / 306386224 / -0.7% 306810098 / 307383831 / -0.2% 0.99 / 1.00 / -0.5%
compress time/wide table cols=100 chunks=1 rows=1000 3862116 / 3748931 / +3.0% 4383702 / 4398059 / -0.3% 0.88 / 0.85 / +3.4%
compress time/wide table cols=100 chunks=50 rows=1000 3671276 / 3743644 / -1.9% 4239354 / 4546978 / -6.8% 0.87 / 0.82 / +5.2%
compress time/wide table cols=1000 chunks=1 rows=1000 37471146 / 37707725 / -0.6% 38713400 / 39201274 / -1.2% 0.97 / 0.96 / +0.6%
compress time/wide table cols=1000 chunks=50 rows=1000 36696495 / 37203558 / -1.4% 37857325 / 39017799 / -3.0% 0.97 / 0.95 / +1.7%
compress time/wide table cols=10000 chunks=1 rows=1000 375807016 / 377630973 / -0.5% 383852095 / 389773560 / -1.5% 0.98 / 0.97 / +1.1%
compress time/wide table cols=10000 chunks=50 rows=1000 375042571 / 378613433 / -0.9% 380558602 / 384293384 / -1.0% 0.99 / 0.99 / +0.0%
decompress time/arade 54502322 / 54601782 / -0.2% 54546184 / 54859370 / -0.6% 1.00 / 1.00 / +0.4%
decompress time/bimbo 92260308 / 92318156 / -0.1% 92435747 / 92679138 / -0.3% 1.00 / 1.00 / +0.2%
decompress time/cmsprovider 103126963 / 103251782 / -0.1% 102952738 / 103175716 / -0.2% 1.00 / 1.00 / +0.1%
decompress time/euro2016 159045108 / 159426388 / -0.2% 159338726 / 159723065 / -0.2% 1.00 / 1.00 / +0.0%
decompress time/food 26509995 / 26602534 / -0.3% 26637836 / 26742726 / -0.4% 1.00 / 0.99 / +0.0%
decompress time/hashtags 446360982 / 445133732 / +0.3% 446829438 / 445350725 / +0.3% 1.00 / 1.00 / -0.1%
decompress time/sparse-wide-flat-runs 14839996 / 15276262 / -2.9% 15454665 / 15883138 / -2.7% 0.96 / 0.96 / -0.2%
decompress time/sparse-wide-lists-runs 277026278 / 277390748 / -0.1% 277115417 / 278014265 / -0.3% 1.00 / 1.00 / +0.2%
decompress time/sparse-wide-lists-shuffled 318406584 / 319073929 / -0.2% 318938616 / 319901475 / -0.3% 1.00 / 1.00 / +0.1%
decompress time/sparse-wide-nested-child-nulls 52829065 / 52809850 / +0.0% 53048064 / 53137390 / -0.2% 1.00 / 0.99 / +0.2%
decompress time/sparse-wide-nested-runs 52853718 / 52843989 / +0.0% 53089138 / 53169078 / -0.2% 1.00 / 0.99 / +0.2%
decompress time/statpopgen 2906614193 / 2928301498 / -0.7% 2934629309 / 2992571106 / -1.9% 0.99 / 0.98 / +1.2%
decompress time/taxi 60568122 / 60795858 / -0.4% 60678810 / 61049394 / -0.6% 1.00 / 1.00 / +0.2%
decompress time/tpc-h l_comment canonical 35987374 / 35994949 / -0.0% 36163646 / 36091820 / +0.2% 1.00 / 1.00 / -0.2%
decompress time/tpc-h l_comment chunked 35963100 / 35999986 / -0.1% 36058355 / 36126936 / -0.2% 1.00 / 1.00 / +0.1%
decompress time/tpc-h sf1 lineitem 82084593 / 82255822 / -0.2% 82625039 / 82329605 / +0.4% 0.99 / 1.00 / -0.6%
decompress time/tpc-h sf1 orders 87810394 / 88171542 / -0.4% 88068704 / 88469438 / -0.5% 1.00 / 1.00 / +0.0%
decompress time/tpc-h sf1 partsupp 111000402 / 111201218 / -0.2% 111095889 / 111755942 / -0.6% 1.00 / 1.00 / +0.4%
decompress time/wide table cols=100 chunks=1 rows=1000 1940060 / 1956334 / -0.8% 2145266 / 2083166 / +3.0% 0.90 / 0.94 / -3.7%
decompress time/wide table cols=100 chunks=50 rows=1000 1946734 / 1950982 / -0.2% 2088892 / 2112020 / -1.1% 0.93 / 0.92 / +0.9%
decompress time/wide table cols=1000 chunks=1 rows=1000 22446560 / 22819352 / -1.6% 23297433 / 23784360 / -2.0% 0.96 / 0.96 / +0.4%
decompress time/wide table cols=1000 chunks=50 rows=1000 22137256 / 22613362 / -2.1% 22763443 / 23761161 / -4.2% 0.97 / 0.95 / +2.2%
decompress time/wide table cols=10000 chunks=1 rows=1000 231972080 / 233845626 / -0.8% 232912425 / 236270815 / -1.4% 1.00 / 0.99 / +0.6%
decompress time/wide table cols=10000 chunks=50 rows=1000 230829521 / 234363140 / -1.5% 232055943 / 236747868 / -2.0% 0.99 / 0.99 / +0.5%
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
parquet size/arade 210020851 / 210020851 / +0.0%
parquet size/bimbo 376187078 / 376187078 / +0.0%
parquet size/cmsprovider 364268159 / 364268159 / +0.0%
parquet size/euro2016 111035158 / 111035158 / +0.0%
parquet size/food 34903783 / 34903783 / +0.0%
parquet size/hashtags 126189203 / 126189203 / +0.0%
parquet size/sparse-wide-flat-runs 1388944 / 1388944 / +0.0%
parquet size/sparse-wide-lists-runs 4327813 / 4327813 / +0.0%
parquet size/sparse-wide-lists-shuffled 5710569 / 5710569 / +0.0%
parquet size/sparse-wide-nested-child-nulls 1266042 / 1266042 / +0.0%
parquet size/sparse-wide-nested-runs 1388944 / 1388944 / +0.0%
parquet size/statpopgen 81284006 / 81284006 / +0.0%
parquet size/taxi 52142534 / 52142534 / +0.0%
parquet size/tpc-h l_comment canonical 52204116 / 52204116 / +0.0%
parquet size/tpc-h l_comment chunked 52204116 / 52204116 / +0.0%
parquet size/tpc-h sf1 lineitem 153403836 / 153403836 / +0.0%
parquet size/tpc-h sf1 orders 37029542 / 37029542 / +0.0%
parquet size/tpc-h sf1 partsupp 29200612 / 29200612 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000 93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000 93240004 / 93240004 / +0.0%
vortex / arrow-ipc / ns (0.992x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 94869939 / 95865668 / -1.0% 119887581 / 122415926 / -2.1% 0.79 / 0.78 / +1.0%
compress time/bimbo 593566720 / 603397805 / -1.6% 655008070 / 668158410 / -2.0% 0.91 / 0.90 / +0.3%
compress time/cmsprovider 376092902 / 380564888 / -1.2% 437860778 / 445103703 / -1.6% 0.86 / 0.86 / +0.5%
compress time/euro2016 38095784 / 38319792 / -0.6% 51213993 / 52782696 / -3.0% 0.74 / 0.73 / +2.5%
compress time/food 39782942 / 40343196 / -1.4% 52178083 / 54048118 / -3.5% 0.76 / 0.75 / +2.1%
compress time/hashtags 77363708 / 78276669 / -1.2% 104645984 / 106901128 / -2.1% 0.74 / 0.73 / +1.0%
compress time/sparse-wide-flat-runs 27891848 / 27922224 / -0.1% 36688200 / 37253526 / -1.5% 0.76 / 0.75 / +1.4%
compress time/sparse-wide-lists-runs 44798466 / 45075541 / -0.6% 51173690 / 51993300 / -1.6% 0.88 / 0.87 / +1.0%
compress time/sparse-wide-lists-shuffled 46130072 / 46588675 / -1.0% 49597461 / 50386218 / -1.6% 0.93 / 0.92 / +0.6%
compress time/sparse-wide-nested-child-nulls 40633090 / 40864326 / -0.6% 50191002 / 50936751 / -1.5% 0.81 / 0.80 / +0.9%
compress time/sparse-wide-nested-runs 40650484 / 40676459 / -0.1% 50159627 / 50782898 / -1.2% 0.81 / 0.80 / +1.2%
compress time/statpopgen 215328764 / 216330328 / -0.5% 261150766 / 262890095 / -0.7% 0.82 / 0.82 / +0.2%
compress time/taxi 58429718 / 59255102 / -1.4% 74800283 / 76397935 / -2.1% 0.78 / 0.78 / +0.7%
compress time/tpc-h l_comment canonical 22503062 / 22571036 / -0.3% 30391995 / 30895072 / -1.6% 0.74 / 0.73 / +1.3%
compress time/tpc-h l_comment chunked 22329973 / 22513906 / -0.8% 30897569 / 30838197 / +0.2% 0.72 / 0.73 / -1.0%
compress time/tpc-h sf1 lineitem 128405014 / 128575520 / -0.1% 162720711 / 165030849 / -1.4% 0.79 / 0.78 / +1.3%
compress time/tpc-h sf1 orders 24548117 / 24682734 / -0.5% 33530984 / 34551466 / -3.0% 0.73 / 0.71 / +2.5%
compress time/tpc-h sf1 partsupp 13076223 / 13158730 / -0.6% 18537967 / 19405427 / -4.5% 0.71 / 0.68 / +4.0%
compress time/wide table cols=100 chunks=1 rows=1000 195875 / 206229 / -5.0% 417312 / 652732 / -36.1% 🟢 0.47 / 0.32 / +48.6%
compress time/wide table cols=100 chunks=50 rows=1000 198882.5 / 199623.5 / -0.4% 372579 / 622612 / -40.2% 🟢 0.53 / 0.32 / +66.5%
compress time/wide table cols=1000 chunks=1 rows=1000 2322270 / 2341174 / -0.8% 2815263 / 3186366 / -11.6% 🟢 0.82 / 0.73 / +12.3%
compress time/wide table cols=1000 chunks=50 rows=1000 2306416 / 2320939 / -0.6% 2781229 / 3082409 / -9.8% 0.83 / 0.75 / +10.1%
compress time/wide table cols=10000 chunks=1 rows=1000 24818310 / 24878252 / -0.2% 30908240 / 31350770 / -1.4% 0.80 / 0.79 / +1.2%
compress time/wide table cols=10000 chunks=50 rows=1000 24862998 / 24957414 / -0.4% 31032220 / 31501179 / -1.5% 0.80 / 0.79 / +1.1%
decompress time/arade 216887782 / 210239864 / +3.2% 217124647 / 210426535 / +3.2% 1.00 / 1.00 / -0.0%
decompress time/bimbo 395863183 / 424705086 / -6.8% 403058101 / 424732118 / -5.1% 0.98 / 1.00 / -1.8%
decompress time/cmsprovider 671764070 / 674213448 / -0.4% 672463177 / 674817862 / -0.3% 1.00 / 1.00 / -0.0%
decompress time/euro2016 83480716 / 83465004 / +0.0% 83746715 / 83689290 / +0.1% 1.00 / 1.00 / -0.0%
decompress time/food 67094070 / 67242871 / -0.2% 67258316 / 67343455 / -0.1% 1.00 / 1.00 / -0.1%
decompress time/hashtags 162082129 / 160347404 / +1.1% 162439484 / 160758461 / +1.0% 1.00 / 1.00 / +0.0%
decompress time/sparse-wide-flat-runs 18537984 / 18747056 / -1.1% 19218870 / 19847949 / -3.2% 0.96 / 0.94 / +2.1%
decompress time/sparse-wide-lists-runs 81221672 / 81718661 / -0.6% 81307964 / 81853158 / -0.7% 1.00 / 1.00 / +0.1%
decompress time/sparse-wide-lists-shuffled 81637744 / 81934875 / -0.4% 81748012 / 82021176 / -0.3% 1.00 / 1.00 / -0.0%
decompress time/sparse-wide-nested-child-nulls 26375638 / 26583464 / -0.8% 28018033 / 27717418 / +1.1% 0.94 / 0.96 / -1.8%
decompress time/sparse-wide-nested-runs 26645024 / 26382980 / +1.0% 28942914 / 26868869 / +7.7% 0.92 / 0.98 / -6.2%
decompress time/statpopgen 459441418 / 469823662 / -2.2% 459818784 / 470310750 / -2.2% 1.00 / 1.00 / +0.0%
decompress time/taxi 51075614 / 50900952 / +0.3% 51378182 / 51131487 / +0.5% 0.99 / 1.00 / -0.1%
decompress time/tpc-h l_comment canonical 40843366 / 40921581 / -0.2% 41161127 / 41228932 / -0.2% 0.99 / 0.99 / -0.0%
decompress time/tpc-h l_comment chunked 40881221 / 40857360 / +0.1% 41146850 / 41224683 / -0.2% 0.99 / 0.99 / +0.2%
decompress time/tpc-h sf1 lineitem 188105094 / 191759513 / -1.9% 188436934 / 191823063 / -1.8% 1.00 / 1.00 / -0.1%
decompress time/tpc-h sf1 orders 38291006 / 39546924 / -3.2% 38338343 / 39636637 / -3.3% 1.00 / 1.00 / +0.1%
decompress time/tpc-h sf1 partsupp 13567490 / 13864499 / -2.1% 13683531 / 14039661 / -2.5% 0.99 / 0.99 / +0.4%
decompress time/wide table cols=100 chunks=1 rows=1000 316725.5 / 317044 / -0.1% 343358 / 338859 / +1.3% 0.92 / 0.94 / -1.4%
decompress time/wide table cols=100 chunks=50 rows=1000 316345.5 / 316698 / -0.1% 339722 / 341353 / -0.5% 0.93 / 0.93 / +0.4%
decompress time/wide table cols=1000 chunks=1 rows=1000 3503534 / 3528793 / -0.7% 3566368 / 3784174 / -5.8% 0.98 / 0.93 / +5.3%
decompress time/wide table cols=1000 chunks=50 rows=1000 3704763 / 3783188 / -2.1% 3858736 / 3776298 / +2.2% 0.96 / 1.00 / -4.2%
decompress time/wide table cols=10000 chunks=1 rows=1000 36147546 / 35619092 / +1.5% 36567067 / 36139101 / +1.2% 0.99 / 0.99 / +0.3%
decompress time/wide table cols=10000 chunks=50 rows=1000 36316956 / 36336174 / -0.1% 36685967 / 36983653 / -0.8% 0.99 / 0.98 / +0.8%
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
arrow-ipc size/arade 680639722 / 680639722 / +0.0%
arrow-ipc size/bimbo 3168319202 / 3168319202 / +0.0%
arrow-ipc size/cmsprovider 2394412322 / 2394412322 / +0.0%
arrow-ipc size/euro2016 359300042 / 359300042 / +0.0%
arrow-ipc size/food 252069442 / 252069442 / +0.0%
arrow-ipc size/hashtags 700364562 / 700364562 / +0.0%
arrow-ipc size/sparse-wide-flat-runs 245220370 / 245220370 / +0.0%
arrow-ipc size/sparse-wide-lists-runs 142046306 / 142046306 / +0.0%
arrow-ipc size/sparse-wide-lists-shuffled 142870882 / 142870882 / +0.0%
arrow-ipc size/sparse-wide-nested-child-nulls 251527506 / 251527506 / +0.0%
arrow-ipc size/sparse-wide-nested-runs 251527122 / 251527122 / +0.0%
arrow-ipc size/statpopgen 1890451826 / 1890451826 / +0.0%
arrow-ipc size/taxi 474481242 / 474481242 / +0.0%
arrow-ipc size/tpc-h l_comment canonical 184829074 / 184829074 / +0.0%
arrow-ipc size/tpc-h l_comment chunked 184829074 / 184829074 / +0.0%
arrow-ipc size/tpc-h sf1 lineitem 1021072290 / 1021072290 / +0.0%
arrow-ipc size/tpc-h sf1 orders 194812026 / 194812026 / +0.0%
arrow-ipc size/tpc-h sf1 partsupp 131286562 / 131286562 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000 1231690 / 1231690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000 1231690 / 1231690 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000 12312506 / 12312506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000 12312506 / 12312506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000 123192506 / 123192506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000 123192506 / 123192506 / +0.0%

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn added the changelog/performance A performance improvement label Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant