Skip to content

Experiment with retained aggregate states in compression - #10180

Closed
connortsui20 wants to merge 1 commit into
ct/aggregate-input-ownerfrom
ct/aggregate-compressor-stats
Closed

connortsui20 wants to merge 1 commit into
ct/aggregate-input-ownerfrom
ct/aggregate-compressor-stats

Conversation

@connortsui20

@connortsui20 connortsui20 commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Archived compressor prototype while the experiment is simplified. The original implementation and benchmark evidence remain preserved.

Original PR description

Progress towards #10175

Summary

The compressor already reuses analysis across candidate schemes, but its hardcoded statistics bundles remain separate from aggregate computation. This experiment puts that reuse under an explicit input lifetime while retaining the native maps, sets, and run information used to construct encodings.

Depends on #10179. Promotion of this ownership model remains an open question in the Epic.

Changes

Schemes use bound aggregate requirements and borrowed retained states in place of ArrayAndStats. Distribution states preserve dictionary preparation and fused run analysis, and samples and derived inputs receive fresh owners.

Physical string fingerprints remain representation-specific. Existing floating-point run semantics and encoding heuristics are preserved so the experiment can evaluate ownership without combining semantic changes.

API Changes

Compressor extensions use ArrayInput and aggregate requirements instead of the legacy statistics bundles. Rich retained states are opt-in and remain separate from finalized file summaries.

@connortsui20 connortsui20 added the changelog/break A breaking API change label Oct 1, 2026
@connortsui20
connortsui20 added this pull request to stack #10181 October 1, 2026 16:05
@connortsui20
connortsui20 force-pushed the ct/aggregate-input-owner branch from 7b210bd to 7d17dc1 Compare October 1, 2026 16:13
@connortsui20
connortsui20 force-pushed the ct/aggregate-compressor-stats branch from 4412cb4 to 166c0dd Compare October 1, 2026 16:13
@codspeed

codspeed Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 26.58%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 1 regressed benchmark
✅ 2288 untouched benchmarks
⏩ 503 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation bench_compare_varbinview[(10000, 2048)] 152.6 µs 207.8 µs -26.58%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/aggregate-compressor-stats (f9ea7ff) with ct/aggregate-input-owner (34b84b9)

Open in CodSpeed

Footnotes

  1. 503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@connortsui20
connortsui20 force-pushed the ct/aggregate-input-owner branch from 7d17dc1 to 6de2d57 Compare October 1, 2026 16:18
@connortsui20
connortsui20 force-pushed the ct/aggregate-compressor-stats branch from 166c0dd to 0a3139f Compare October 1, 2026 16:18
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/aggregate-input-owner branch from 6de2d57 to 34b84b9 Compare October 1, 2026 18:07
@connortsui20
connortsui20 force-pushed the ct/aggregate-compressor-stats branch from 0a3139f to f9ea7ff Compare October 1, 2026 18:07
@connortsui20
connortsui20 removed this pull request from stack #10181 October 5, 2026 10:43
@connortsui20
connortsui20 deleted the ct/aggregate-compressor-stats branch October 5, 2026 10:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant