Skip to content

fix(compressor): amortize fixed table bytes in sampled estimates - #11402

Open
robert3005 wants to merge 2 commits into
developfrom
claude/quirky-bohr-2mpgog
Open

robert3005 wants to merge 2 commits into
developfrom
claude/quirky-bohr-2mpgog

Conversation

@robert3005

@robert3005 robert3005 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

On unique text such as TPC-H o_comment, OnPair is 25–38% smaller than FSST on every chunk, yet the compressor picked FSST on about half of them. The ~1,000-row sample used for scheme selection put the two at almost the same size, so the winner was a coin flip per chunk.

The sample penalises OnPair in two ways:

  • Its dictionary is charged in full. The dictionary is capped at 4,096 tokens whatever the row count. On a 1,024-row sample of a 24,576-row chunk it is 4.2 KB of a 20.7 KB output (20%); at full size it is a few percent. FSST's symbol table is at most about 2.3 KB, so it pays far less.
  • It trains a worse dictionary. The trainer scans 15% of its input: about 3,700 rows of the full chunk but only about 150 rows of the sample.

Changes

  • Scheme::fixed_nbytes (new, defaults to 0): a scheme reports the output bytes that do not grow with row count, and the sampler charges only the share the sample makes up of the full array. OnPair reports its dictionary, FSST its symbol table.
  • When compressing a sample, OnPair trains on half of it instead of 15%. The 0.5 is empirical: scanning the whole sample made random text pick OnPair, which is about 4% larger there.

Measured on 8 chunks each of 8,192, 24,576 and 65,536 rows, before the rebase onto #9762: o_comment, l_comment and synthetic URLs now pick OnPair on every chunk, and random hex and random printable text pick FSST, which is the smaller scheme in each case.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants