Repository navigation
Remove unused unbound expression analysis and traversal helpers - #9993
Conversation
Merging this PR will improve performance by 21.56%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | mul_u64_nonnull_neon |
40 µs | 28.5 µs | +40.15% |
| ⚡ | WallTime | mul_i64_nonnull_neon |
38.7 µs | 32.5 µs | +18.96% |
| ⚡ | WallTime | multiply_shapes_neon[(32768, PerRowPerRow)] |
38.7 µs | 32.6 µs | +18.91% |
| ⚡ | Simulation | sum_v2_i64 |
224.6 µs | 193.9 µs | +15.87% |
| ⚡ | Simulation | sum_i64 |
223.7 µs | 193.6 µs | +15.56% |
| Simulation | fixed_16_advancing_ptr_safe[100] |
< 1 ns | < 1 ns | N/A | |
| Simulation | preverify_advancing_ptr_unchecked[1000] |
< 1 ns | < 1 ns | N/A | |
| Simulation | preverify_advancing_ptr_unchecked[10000] |
< 1 ns | < 1 ns | N/A |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/remove-unused-unbound-expressions (33b0e9f) with develop (bc9a579)
Footnotes
-
479 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.990x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.064x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.038x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.024x ➖, 0↑ 0↓)
File Size Changes (100 files changed, +0.0% overall, 53↑ 47↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.992x ➖, 1↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.001x ➖, 1↑ 2↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 1↑ 7↓)
duckdb / parquet / ns (0.996x ➖, 3↑ 4↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.990x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.849x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.672x ✅, 5↑ 0↓)
duckdb / vortex-file-compressed / ns (0.857x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.796x ➖, 1↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.974x ➖, 1↑ 3↓)
datafusion / parquet / ns (0.821x ➖, 2↑ 0↓)
duckdb / vortex-file-compressed / ns (0.955x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.830x ➖, 2↑ 0↓)
|
Supersedes #9514. Optimization now runs only on `BoundExpression`s: callers bind first, then optimize. This removes the optimizer-owned dtype cache, rules can read dtypes straight off the tree, and it unblocks lambdas/variables, which must be typed before optimization. ## Changes - `BoundExpression::optimize` and `optimize_recursive` replace the `Expression` versions. Traversal and the fixpoint loop are unchanged. - `ScalarFnVTable::simplify` now takes a `&BoundExpression`, and `simplify_untyped` is merged into it. `SimplifyCtx` and `SimplifyCache` are removed. - `ExpressionReduceNode` keeps its name and `reduce_expression` dispatch, but now borrows a bound tree. - Rules ported: binary, case_when, cast, fill_null, mask, select, zip, and `find_between`. `GetItem::simplify_untyped` is removed because it duplicated `GetItem::reduce`. If the rebuilt `Between` fails to type-check, `find_between` now skips the rewrite instead of producing a tree that fails to bind. - Callers change from `expr.optimize_recursive(dtype)?.bind(dtype)` to `expr.bind(dtype)?.optimize_recursive()`. - The unbound `expr::transform::partition` is deleted. It was the last user of the unbound optimizer and has no callers; layouts use `bound_partition`. #9993 deletes it too, so whichever PR lands second will have a trivial conflict. ## API changes - `Expression::optimize(scope)` / `optimize_recursive(scope)` become `BoundExpression::optimize()` / `optimize_recursive()`. - `ScalarFnVTable::simplify(options, &BoundExpression)` replaces `simplify(options, &Expression, &dyn SimplifyCtx)` and `simplify_untyped`. - `ExpressionReduceNode::new(&BoundExpression)` drops the scope argument. - `partition`, `partition_annotations` and `PartitionedExpr` are removed. ## Behavior I checked this with a differential corpus of 30,018 generated expressions targeting every rule, comparing develop's `optimize + bind` against the new `bind + optimize` and evaluating before/after on data: - 28,878 produce identical output. - 131 panicked on develop (`Mask` simplify called `vortex_expect` on a null mask literal) and now fail cleanly at bind time. - 941 have ill-typed inputs that develop accepted only because a rule removed the ill-typed subtree before binding. They now error. - 68 differ because of a develop bug. `SimplifyCache` keys dtypes by `Expression`, and `Scalar` equality ignores nullability, so develop reused the wrong cached dtype and dropped `fill_null`s it should have kept. No well-typed expression regressed. Existing bugs found, not addressed here: - The `CASE WHEN`→`fill_null` rewrite is unexecutable for utf8. - `select([], nullable_struct)` drops the struct's validity. - `CaseWhen` execution can return a result that is non-nullable when its dtype is nullable. - `and`/`or`/`zip` literal rules can change the result's nullability. ## Performance The optimizer benchmarks bind outside the timed loop and measure only `optimize_recursive`, the same work develop's benchmarks time. Local medians: | case | develop | this PR | |---|---|---| | `or_chain` 200 | 17.5 µs | 18.6 µs | | `balanced_or` 64–256 ids | 6.2–23.0 µs | 6.1–23.5 µs | | `balanced_or` 1 id | 0.79 µs | 0.94 µs | | `in_list` | ~0.77 µs | ~0.95 µs | Small predicates pay a constant ~0.15–0.2 µs in the optimizer, almost all of it in `find_between`, which now type-checks the `and`/`Between` nodes it rebuilds (0.56 µs vs 0.34 µs). Across the full `bind + optimize` pipeline that callers run, the gap is ~0.3–0.45 µs, because binding also types the comparisons `find_between` later replaces. Closing it would need an unchecked `BoundExpression` constructor. Open question. ## Testing - `cargo nextest run -p vortex-array -p vortex-layout -p vortex-file -p vortex -p vortex-datafusion -p vortex-duckdb` - `cargo test --doc -p vortex -p vortex-array` - `cargo clippy --all-targets --all-features -- -D warnings` on all changed crates and `fuzz/file_io` - `cargo check --workspace --all-targets` Not run: Python tests. `vortex-cuda-ffi` was not type-checked because it is compiled out on macOS. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Delete obsolete unbound partitioning, labeling, and field-reference collection, and remove their unused supporting APIs. Keep the active bound analysis and shared traversal paths, adapting existing tests to the surviving APIs. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
4fbac1f to
33b0e9f
Compare
Summary
Expression analysis and partitioning now use bound expressions in layout planning, leaving the old unbound implementations with no production callers in this repository. Remove those implementations and their supporting APIs,.
Changes
analysis/strict.rs,analysis/infallible.rs,transform/partition.rs, andtraversal/references.rs, including tests specific to the removed APIs.VortexExprExt,ExactExpr,split_conjunction, andreplace_root_fields.API Changes
This removes public Rust APIs with no production callers in the repository: unbound partitioning/analysis helpers and associated types,
ReferenceCollector,VortexExprExt,ExactExpr,split_conjunction,replace_root_fields,NodeFolder,FoldDown,NodeExt::fold,NodeExt::transform,NodeExt::transform_up, andpre_order_visit_up.AnnotationFnnow defaults its node type toBoundExpression.