Skip to content

fix(reader): size prefetch by decoded bytes - #99

Merged
anoop-narang merged 2 commits into
mainfrom
fix/prefetch-decoded-estimate
Nov 18, 2026
Merged

anoop-narang merged 2 commits into
mainfrom
fix/prefetch-decoded-estimate

Conversation

@zfarrell

Copy link
Copy Markdown

The prefetch headroom gate sized row groups by parquet's encoded page size, so dictionary-encoded strings and decimals could pin many times what it allowed for; it now estimates the decoded Arrow size. Closes #80

The headroom gate that decides whether to prefetch the next row group
compared the cache's free memory with parquet's uncompressed_size, the
encoded page size. Dictionary- and RLE-encoded columns decode to many
times that, so a prefetch could pin ~85x more than the gate allowed for.

Estimate each column chunk's decoded Arrow size from its physical type,
using unencoded_byte_array_data_bytes (or min/max lengths) for byte
arrays, and never below the encoded size.

Refs #80
decoded_column_bytes counted INT32/INT64 decimals at 4/8 bytes, but the
reader decodes every decimal to Decimal128 (16 bytes) or, above
precision 38, Decimal256 (32 bytes), whatever its physical type.

Without unencoded_byte_array_data_bytes, string data was sized from the
min/max values alone, which are ordered by value rather than length and
often truncated. Take the larger of that and the encoded size, and
document the result as a best guess that can still under-count.

Refs #80
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.81890% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/datafusion/src/reader/runtime/morsel.rs 91.17% 2 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@github-actions

Copy link
Copy Markdown

📊 Benchmark Comparison

Current: f24d507c (Liquid) vs Baseline: f24d507c (DataFusionDefault)

Query Cold Time Δ Warm Time Δ CPU Time Δ
Q1 2.0ms (2.0ms) +0.0% 0.000ms (0.000ms) +0.0% 0.000ms (0.000ms) +0.0%
Q2 6.0ms (4.0ms) +50.0% 2.0ms (3.0ms) -33.3% 5.0ms (5.0ms) +0.0%
Q3 10.0ms (9.0ms) +11.1% 3.0ms (9.0ms) -66.7% 2.0ms (18.5ms) -89.2%
Q4 11.0ms (9.0ms) +22.2% 3.0ms (7.0ms) -57.1% 2.0ms (15.0ms) -86.7%
Q5 43.0ms (41.0ms) +4.9% 33.0ms (40.0ms) -17.5% 3.0ms (16.5ms) -81.8%
Q6 165.0ms (98.0ms) +68.4% 56.5ms (85.5ms) -33.9% 17.0ms (61.0ms) -72.1%
Q7 1.0ms (1.0ms) +0.0% 0.000ms (0.000ms) +0.0% 0.000ms (0.000ms) +0.0%
Q8 7.0ms (5.0ms) +40.0% 3.5ms (4.5ms) -22.2% 6.0ms (5.0ms) +20.0%
Q9 69.0ms (68.0ms) +1.5% 59.5ms (66.5ms) -10.5% 5.0ms (31.5ms) -84.1%
Q10 84.0ms (73.0ms) +15.1% 54.0ms (71.0ms) -23.9% 5.0ms (48.0ms) -89.6%
Q11 79.0ms (20.0ms) +295.0% 63.5ms (18.0ms) +252.8% 176.5ms (24.5ms) +620.4%
Q12 85.0ms (21.0ms) +304.8% 65.0ms (21.5ms) +202.3% 177.5ms (29.0ms) +512.1%
Q13 246.0ms (92.0ms) +167.4% 117.0ms (92.5ms) +26.5% 122.5ms (62.5ms) +96.0%
Q14 429.0ms (111.0ms) +286.5% 104.0ms (112.5ms) -7.6% 69.5ms (77.5ms) -10.3%
Q15 334.0ms (84.0ms) +297.6% 111.5ms (90.0ms) +23.9% 110.0ms (76.0ms) +44.7%
Q16 90.0ms (89.0ms) +1.1% 84.5ms (89.5ms) -5.6% 4.0ms (16.5ms) -75.8%
Q17 450.0ms (182.0ms) +147.3% 199.0ms (184.0ms) +8.2% 50.0ms (77.5ms) -35.5%
Q18 433.0ms (183.0ms) +136.6% 202.0ms (182.5ms) +10.7% 50.5ms (77.5ms) -34.8%
Q19 667.0ms (392.0ms) +70.2% 313.0ms (356.5ms) -12.2% 71.5ms (115.5ms) -38.1%
Q20 11.0ms (8.0ms) +37.5% 2.5ms (7.0ms) -64.3% 6.0ms (15.0ms) -60.0%
Q21 1.35s (142.0ms) +847.9% 403.0ms (144.5ms) +178.9% 258.0ms (226.5ms) +13.9%
Q22 1.59s (138.0ms) +1052.2% 555.0ms (135.5ms) +309.6% 119.5ms (273.0ms) -56.2%
Q23 3.47s (379.0ms) +815.8% 1.63s (388.5ms) +319.2% 391.0ms (601.5ms) -35.0%
Q24 21.33s (751.0ms) +2739.9% 965.5ms (782.0ms) +23.5% 428.5ms (2.12s) -79.8%
Q25 303.0ms (58.0ms) +422.4% 27.0ms (46.5ms) -41.9% 50.0ms (94.0ms) -46.8%
Q26 168.0ms (35.0ms) +380.0% 47.0ms (36.0ms) +30.6% 121.5ms (62.5ms) +94.4%
Q27 295.0ms (51.0ms) +478.4% 36.0ms (45.0ms) -20.0% 73.5ms (92.5ms) -20.5%
Q28 1.27s (174.0ms) +631.0% 829.5ms (173.5ms) +378.1% 149.5ms (223.0ms) -33.0%
Q29 1.82s (775.0ms) +135.2% 977.5ms (794.5ms) +23.0% 299.0ms (284.5ms) +5.1%
Q30 24.0ms (22.0ms) +9.1% 17.5ms (21.0ms) -16.7% 5.0ms (16.5ms) -69.7%
Q31 545.0ms (86.0ms) +533.7% 75.0ms (81.5ms) -8.0% 71.5ms (112.0ms) -36.2%
Q32 773.0ms (82.0ms) +842.7% 105.0ms (83.0ms) +26.5% 79.0ms (118.0ms) -33.1%
Q33 297.0ms (282.0ms) +5.3% 267.0ms (282.5ms) -5.5% 8.0ms (59.0ms) -86.4%
Q34 1.37s (353.0ms) +288.7% 557.0ms (380.0ms) +46.6% 111.0ms (227.5ms) -51.2%
Q35 1.35s (351.0ms) +284.0% 586.0ms (361.0ms) +62.3% 105.5ms (220.0ms) -52.0%
Q36 87.0ms (85.0ms) +2.4% 73.0ms (83.5ms) -12.6% 4.0ms (17.0ms) -76.5%
Q37 297.0ms (82.0ms) +262.2% 74.0ms (83.5ms) -11.4% 21.5ms (59.0ms) -63.6%
Q38 68.0ms (36.0ms) +88.9% 28.0ms (38.0ms) -26.3% 20.0ms (19.5ms) +2.6%
Q39 263.0ms (41.0ms) +541.5% 23.5ms (41.5ms) -43.4% 12.5ms (63.0ms) -80.2%
Q40 711.0ms (154.0ms) +361.7% 201.0ms (152.0ms) +32.2% 45.5ms (107.5ms) -57.7%
Q41 18.0ms (15.0ms) +20.0% 9.0ms (15.0ms) -40.0% 6.0ms (13.0ms) -53.8%
Q42 20.0ms (16.0ms) +25.0% 8.0ms (14.5ms) -44.8% 6.5ms (13.0ms) -50.0%
Q43 14.0ms (14.0ms) +0.0% 9.5ms (12.0ms) -20.8% 7.5ms (8.0ms) -6.2%

⚠️ LiquidCache is slower on 15 queries (warm)

  • Q28: warm +378.1% (829.5ms vs 173.5ms)
  • Q23: warm +319.2% (1.63s vs 388.5ms)
  • Q22: warm +309.6% (555.0ms vs 135.5ms)
  • Q11: warm +252.8% (63.5ms vs 18.0ms)
  • Q12: warm +202.3% (65.0ms vs 21.5ms)
  • Q21: warm +178.9% (403.0ms vs 144.5ms)
  • Q35: warm +62.3% (586.0ms vs 361.0ms)
  • Q34: warm +46.6% (557.0ms vs 380.0ms)
  • Q40: warm +32.2% (201.0ms vs 152.0ms)
  • Q26: warm +30.6% (47.0ms vs 36.0ms)
  • Q32: warm +26.5% (105.0ms vs 83.0ms)
  • Q13: warm +26.5% (117.0ms vs 92.5ms)
  • Q15: warm +23.9% (111.5ms vs 90.0ms)
  • Q24: warm +23.5% (965.5ms vs 782.0ms)
  • Q29: warm +23.0% (977.5ms vs 794.5ms)

Compared Liquid vs DataFusionDefault on the same runner
Regressions: warm-time increases of at least 15%. Cold Time: first iteration; Warm Time: median of remaining iterations.

@zfarrell
zfarrell marked this pull request as ready for review November 17, 2026 20:35
@zfarrell
zfarrell requested a review from a team as a code owner November 17, 2026 20:35
@zfarrell
zfarrell requested review from anoop-narang and removed request for a team November 17, 2026 20:35
@anoop-narang
anoop-narang merged commit 63fe150 into main Nov 18, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reader: prefetch estimate under-counts decoded strings ~100x

2 participants