You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
reader: does a TopK scan stop early inside a file? #98
Not a confirmed bug. This records a check that has not been done.
cf309e6 (#65) makes an ORDER BY ... LIMIT scan read its best files first: LiquidParquetSource now implements try_pushdown_sort and reorder_files. Its commit message says "Row groups inside a file are still read in file order".
What has not been checked:
whether the reader re-checks the TopK threshold at each row-group boundary, so it can skip row groups whose statistics can no longer contribute, and
whether it stops reading a file early once the threshold rules out the rest of it.
If neither happens, a TopK query still decodes every row group of each file it opens, even when the first one already fills the limit.
To answer this: run a TopK query over one file with several sorted row groups and count the row groups read.
Related: #66. A scan that stops early cancels in-flight inserts, which is what strands eviction victims there. Making TopK stop earlier will exercise that path more often.
Activity