Repository navigation
Replace legacy statistics APIs with aggregate results - #10178
connortsui20 wants to merge 13 commits into
Conversation
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Merging this PR will degrade performance by 12.05%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | null_count_run_end[(10000, 1024, 0.01)] |
6.4 µs | 12.4 µs | -48.75% |
| ❌ | Simulation | null_count_run_end[(32000, 256, 0.01)] |
6.4 µs | 12.4 µs | -48.75% |
| ❌ | Simulation | null_count_run_end[(10000, 256, 0.01)] |
6.4 µs | 12.4 µs | -48.32% |
| ❌ | Simulation | null_count_run_end[(32000, 1024, 0.01)] |
6.4 µs | 12.4 µs | -48.32% |
| ❌ | Simulation | bitpacked_compress_u32 |
36.7 µs | 57.8 µs | -36.48% |
| ❌ | Simulation | sparse_null_count |
62.6 µs | 90.9 µs | -31.11% |
| ❌ | Simulation | max_i32 |
101.5 µs | 134 µs | -24.27% |
| ❌ | Simulation | for_compress_i32 |
95.5 µs | 125.8 µs | -24.13% |
| ❌ | Simulation | bench_compare_sliced_dict_primitive[(2000, 10000)] |
84.8 µs | 108.4 µs | -21.78% |
| ❌ | Simulation | max_i32_nulls_clustered |
130 µs | 160.4 µs | -18.99% |
| ❌ | Simulation | slice_dict_tight_loop[10000] |
685.3 µs | 841.2 µs | -18.53% |
| ❌ | Simulation | compress[(1000, 256)] |
98.1 µs | 119 µs | -17.54% |
| ❌ | Simulation | chunked_varbinview_into_canonical[(10, 100)] |
367.6 µs | 443.5 µs | -17.1% |
| ❌ | Simulation | max_i64 |
159.1 µs | 191.8 µs | -17.08% |
| ❌ | Simulation | bitpacked_decompress_u32 |
51.1 µs | 61.2 µs | -16.51% |
| ❌ | Simulation | compress[(1000, 16)] |
104.4 µs | 123.3 µs | -15.32% |
| ❌ | Simulation | canonicalize_sparse_list[(1024, 17, 8)] |
459.2 µs | 539.4 µs | -14.87% |
| ❌ | Simulation | sequence_decompress_u32 |
35.8 µs | 42 µs | -14.77% |
| ❌ | Simulation | canonicalize_sparse_list[(512, 7, 4)] |
534.3 µs | 625.8 µs | -14.61% |
| ❌ | Simulation | compress[(1000, 4)] |
118 µs | 136.3 µs | -13.48% |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/aggregate-array-stats (f69c359) with develop (900c1f2)
Footnotes
-
461 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Superseded by the consolidated statistics migration in #10235, #10265, and #10269. The original implementation and benchmark evidence remain preserved.
Original PR description
Tracking Issue: #10177
Summary
Aggregate functions already define extensible statistics, but the array cache and file summaries still require a separate fixed set of fields and conversion rules. Replacing those APIs lets computation and consumers share the aggregate contract while preserving array caching, so retiring the legacy API does not depend on deciding where caches should live.
Streaming file summaries also need partial states: the old writer collects finalized chunk sums in a nullable intermediate column, where an overflow result can be skipped or dropped. Accumulating states keeps saturation through finalization.
Changes
For review, start with
stats::aggregations,stats::results, andstats::compat, then follow the file-summary and consumer changes. The migration guide explains the result and partial-state boundary.API Changes
Removes runtime
ArrayStats,StatsSet, and fixedStatAPIs. Array computation and file-summary selection and lookup use aggregate functions, including their options. The Rust API changes without a new wire format or edition requirement.