Repository navigation
Share dictionary values between V2 filters and projections - #11098
joseph-isaacs wants to merge 1 commit into
Conversation
The V2 scan lowered a dictionary layout to `Take(codes, values)` with no `Share` above the values. A filter on a dictionary column is rewritten to `Take(codes, Eval(predicate, values))`, and the projection of the same column stays `Take(codes, values)`. The value cache is keyed by plan pointer, so the two plans cached different things and each decoded the dictionary values separately. V1's dictionary reader shares one decode between them. Lower the values under a `Share`, as its docs and `ExpressionTakeRule` already assume, so the predicate evaluates above the share and the projection reuses the values it canonicalized. Struct-valued dictionaries keep bare values so field selections still push through their pack. On a 3-file ClickBench subset with vortex-compact (Zstd dictionary values), the V2 scan now decodes the same number of values as V1, and Q22, Q24-Q27 run 22-49% faster. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBBD8uUsB4hozGEcuCv Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
commented
Nov 2, 2026
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.002x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.015x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
commented
Nov 2, 2026
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.928x ➖, 4↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.940x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.986x ➖, 0↑ 0↓)
|
commented
Nov 2, 2026
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.993x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
commented
Nov 2, 2026
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.014x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.010x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.985x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.985x ➖, 0↑ 0↓)
|
commented
Nov 2, 2026
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.011x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.114x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (0.989x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.056x ➖, 0↑ 0↓)
|
commented
Nov 2, 2026
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.992x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
commented
Nov 2, 2026
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.957x ➖, 5↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.975x ➖, 4↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
commented
Nov 2, 2026
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.997x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
|
commented
Nov 2, 2026
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.992x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.986x ➖, 0↑ 0↓)
|
commented
Nov 2, 2026
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.117x ➖, 0↑ 3↓)
datafusion / parquet / ns (1.070x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (1.052x ➖, 1↑ 2↓)
duckdb / parquet / ns (1.051x ➖, 1↑ 1↓)
|
commented
Nov 2, 2026
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.972x ➖, 2↑ 1↓)
datafusion / parquet / ns (1.046x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (1.108x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.082x ➖, 1↑ 3↓)
|
commented
Nov 2, 2026
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.009x ➖, 0↑ 0↓)
|
commented
Nov 2, 2026
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.981x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.013x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.005x ➖, 0↑ 0↓)
|
commented
Nov 2, 2026
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.973x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.007x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
commented
Nov 2, 2026
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
|
commented
Nov 2, 2026
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.011x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.008x ➖, 0↑ 1↓)
|
commented
Nov 2, 2026
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.007x ➖, 0↑ 3↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.017x ➖, 0↑ 2↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
|
commented
Nov 2, 2026
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.146x ➖, 0↑ 3↓)
datafusion / parquet / ns (1.058x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.112x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.057x ➖, 2↑ 2↓)
|
Summary
ClickBench Q22–Q27 on DataFusion +
vortex-compactregressed when DataFusion ClickBench moved to the V2 scan in 705911c (#10799). For example, Q23 went from 258 to 653 ms and Q27 from 102 to 173 ms. The default vortex format was unaffected. The same jump appears on ClickBench-sorted at 9085458, when that suite moved to V2.When a query filters and projects the same dictionary column, the V2 scan decodes the dictionary values once for the filter and again for the projection. In compact files those values are Zstd-compressed, so the duplicate decode is expensive. V1 does not have this problem because its dictionary reader keeps one decoded copy per dictionary.
Changes
lower_dictlowered a dictionary toTake(codes, values). A filter on the column is rewritten toTake(codes, Eval(pred, values)), while the projection staysTake(codes, values).ValueCacheis keyed by plan pointer, so the two cached different things and neither reused the other's decoded values.Sharewas built for exactly this, andExpressionTakeRulealready keeps predicates above it, but nothing ever created aShare.lower_dictnow puts the values under aShare. Because lazy plan children are memoized and the optimizer keeps the pointer of any subtree it doesn't rewrite, the filter and the projection now point at the sameShareand decode the values once. Struct-valued dictionaries keep bare values so field selections still push through their pack.Local results on 3 ClickBench files (4 cores):
Q23 is still unresolved. The sparse-take branch in
operators/take.rsre-decodes whole Zstd dictionaries per batch: V2 decodes 5.0M values against V1's 3.6M. Disabling that branch hurts FSST, so fixing it properly needs an encoding-aware condition. That is left for a follow-up. Q23's 2.6× regression in CI also did not reproduce at 3 files, which is part of why this PR asks for the CI benchmark.Tests:
predicate_and_projection_share_dictionary_valuestest inplan/physical/pipeline/tests.rs. It fails without the fix (the projection reads the values again) and passes with it.sharenode.cargo test -p vortex-layout --lib(736 passed),cargo test -p vortex-datafusion(788 + 14 passed), andcargo clippy -p vortex-layout --all-targets --all-features -- -D warningsare all clean.Generated by Claude Code