Skip to content

Add generic dynamic expressions with generation tracking - #10865

Merged
gatesn merged 1 commit into
developfrom
ngates/dynamic-expression-core
Oct 27, 2026
Merged

gatesn merged 1 commit into
developfrom
ngates/dynamic-expression-core

Conversation

@gatesn

@gatesn gatesn commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Extract the generic predicate source and per-generation bindings from #10665. Reuse native dynamic-expression update tracking for predicates whose declared inputs remain stable.

DataFusion provides entirely dynamic expression trees, while our existing logic only handles dynamic comparisons with dynamic scalars.

This PR adds initial support for dynamic expression trees.

Extract the generic predicate source and per-generation bindings from #10665. Reuse native dynamic-expression update tracking for predicates whose declared inputs remain stable.

Co-authored-by: Robert Kruszewski <github@robertk.io>

Source: #10665 (19a26fd)
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@codspeed

codspeed Bot commented Oct 27, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 23.35%

3 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 2 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

⚡ 8 improved benchmarks
✅ 2196 untouched benchmarks
⏩ 370 skipped benchmarks1
🗄️ 30 archived benchmarks run2

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ CPU Simulation filter_powerlaw_by_random[10000] 765.6 ns 466.9 ns +63.98%
⚡ CPU Simulation missing_stats[16] 88.6 µs 70.9 µs +24.99%
⚡ CPU Simulation bench_compare_varbin[(50000, 2)] 166 µs 134.5 µs +23.42%
⚡ CPU Simulation bench_compare_varbin[(50000, 64)] 167.7 µs 136.4 µs +22.92%
⚡ CPU Simulation bench_compare_sliced_dict_varbinview[(7500, 10000)] 231.9 µs 195.8 µs +18.44%
⚡ CPU Simulation bench_compare_sliced_dict_varbinview[(1000, 10000)] 93.3 µs 80.7 µs +15.65%
⚡ CPU Simulation bench_compare_sliced_dict_varbinview[(10000, 10000)] 237.8 µs 207.9 µs +14.37%
⚡ CPU Simulation bench_compare_sliced_dict_primitive[(2000, 10000)] 76 µs 69.1 µs +10%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ngates/dynamic-expression-core (12ce651) with develop (a5f574e)

Open in CodSpeed

Footnotes

  1. 370 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. 30 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

@gatesn gatesn added the changelog/feature A new feature label Oct 27, 2026
@gatesn
gatesn marked this pull request as ready for review October 27, 2026 15:39
@gatesn
gatesn enabled auto-merge (squash) October 27, 2026 15:39
@gatesn
gatesn merged commit 9d0b790 into develop Oct 27, 2026
109 of 110 checks passed
@gatesn
gatesn deleted the ngates/dynamic-expression-core branch October 27, 2026 15:41
joseph-isaacs added a commit to vortex-data/vortex-datafusion-cli that referenced this pull request Nov 12, 2026
Bumps `vortex-datafusion` from 0.90.0 to 0.93.0. DataFusion stays at
55.2.0, which satisfies 0.93.0's `datafusion ^55.0.0` requirement. The
release tag will be `0.93.0-55.2.0`.

This picks up:
- **0.91.0**: generic dynamic expressions with generation tracking
(vortex-data/vortex#10865), live statistics proofs for them (#10869),
skipping file filters already proven true (#10894), and pushing
DataFusion dynamic filters through Scan API expression trees (#10895)
- **0.92.0**: DataFusion dynamic filters pushed into V2 scans (#11215)
- **0.93.0**: `array_transform` pushdown (#11031)

Note: `scan_v2` still defaults to `false` in 0.93.0, so the V2-only
dynamic filter path needs `OPTIONS (scan_v2 'true')` or `SET
vortex.scan_v2 = true`.

Checked locally: `cargo fmt --check` and `cargo check --locked` pass.
The tests were not run locally; CI runs them.

Supersedes #19.

---
_Generated by [Claude
Code](https://claude.ai/code/session_01UeniBCmx9HLCPXaaVKUcL2)_

Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants