Skip to content

Replace legacy statistics APIs with aggregate results - #10178

Closed
connortsui20 wants to merge 13 commits into
developfrom
ct/aggregate-array-stats
Closed

connortsui20 wants to merge 13 commits into
developfrom
ct/aggregate-array-stats

Conversation

@connortsui20

@connortsui20 connortsui20 commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Superseded by the consolidated statistics migration in #10235, #10265, and #10269. The original implementation and benchmark evidence remain preserved.

Original PR description

Tracking Issue: #10177

Summary

Aggregate functions already define extensible statistics, but the array cache and file summaries still require a separate fixed set of fields and conversion rules. Replacing those APIs lets computation and consumers share the aggregate contract while preserving array caching, so retiring the legacy API does not depend on deciding where caches should live.

Streaming file summaries also need partial states: the old writer collects finalized chunk sums in a nullable intermediate column, where an overflow result can be skipped or dropped. Accumulating states keeps saturation through finalization.

Changes

  • Array metadata holds finalized aggregate results keyed by the function and options. Streaming consumers retain partial states separately.
  • File summaries and engine integrations use the same result API. Compatibility adapters preserve the existing node, footer, and legacy zone formats.
  • Compute and encoding callers move to aggregate requests. Constructor validation continues to check physical invariants independently of generic metadata.

For review, start with stats::aggregations, stats::results, and stats::compat, then follow the file-summary and consumer changes. The migration guide explains the result and partial-state boundary.

API Changes

Removes runtime ArrayStats, StatsSet, and fixed Stat APIs. Array computation and file-summary selection and lookup use aggregate functions, including their options. The Rust API changes without a new wire format or edition requirement.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@codspeed

codspeed Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 12.05%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 10 improved benchmarks
❌ 28 regressed benchmarks
✅ 2101 untouched benchmarks
⏩ 461 skipped benchmarks1
🗄️ 1 archived benchmark run2

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation null_count_run_end[(10000, 1024, 0.01)] 6.4 µs 12.4 µs -48.75%
❌ Simulation null_count_run_end[(32000, 256, 0.01)] 6.4 µs 12.4 µs -48.75%
❌ Simulation null_count_run_end[(10000, 256, 0.01)] 6.4 µs 12.4 µs -48.32%
❌ Simulation null_count_run_end[(32000, 1024, 0.01)] 6.4 µs 12.4 µs -48.32%
❌ Simulation bitpacked_compress_u32 36.7 µs 57.8 µs -36.48%
❌ Simulation sparse_null_count 62.6 µs 90.9 µs -31.11%
❌ Simulation max_i32 101.5 µs 134 µs -24.27%
❌ Simulation for_compress_i32 95.5 µs 125.8 µs -24.13%
❌ Simulation bench_compare_sliced_dict_primitive[(2000, 10000)] 84.8 µs 108.4 µs -21.78%
❌ Simulation max_i32_nulls_clustered 130 µs 160.4 µs -18.99%
❌ Simulation slice_dict_tight_loop[10000] 685.3 µs 841.2 µs -18.53%
❌ Simulation compress[(1000, 256)] 98.1 µs 119 µs -17.54%
❌ Simulation chunked_varbinview_into_canonical[(10, 100)] 367.6 µs 443.5 µs -17.1%
❌ Simulation max_i64 159.1 µs 191.8 µs -17.08%
❌ Simulation bitpacked_decompress_u32 51.1 µs 61.2 µs -16.51%
❌ Simulation compress[(1000, 16)] 104.4 µs 123.3 µs -15.32%
❌ Simulation canonicalize_sparse_list[(1024, 17, 8)] 459.2 µs 539.4 µs -14.87%
❌ Simulation sequence_decompress_u32 35.8 µs 42 µs -14.77%
❌ Simulation canonicalize_sparse_list[(512, 7, 4)] 534.3 µs 625.8 µs -14.61%
❌ Simulation compress[(1000, 4)] 118 µs 136.3 µs -13.48%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/aggregate-array-stats (f69c359) with develop (900c1f2)

Open in CodSpeed

Footnotes

  1. 461 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. 1 benchmark was run, but is now archived. If it was deleted in another branch, consider rebasing to remove it from the report. Instead if it was added back, click here to restore it. ↩

@connortsui20
connortsui20 added this pull request to stack #10181 October 1, 2026 16:05
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@github-actions
github-actions Bot temporarily deployed to docs-preview/pr-10178 October 1, 2026 16:30 Inactive
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@github-actions
github-actions Bot temporarily deployed to docs-preview/pr-10178 October 1, 2026 18:15 Inactive
@connortsui20
connortsui20 removed this pull request from stack #10181 October 5, 2026 10:43
@connortsui20
connortsui20 deleted the ct/aggregate-array-stats branch October 5, 2026 10:44

This branch was previously deployed

1 inactive deployment
docs-preview/pr-10178 — f69c359e Deployed Oct 1, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant