Skip to content

reader: does a TopK scan stop early inside a file? #98

Open

Description

@zfarrell

Not a confirmed bug. This records a check that has not been done.

cf309e6 (#65) makes an ORDER BY ... LIMIT scan read its best files first: LiquidParquetSource now implements try_pushdown_sort and reorder_files. Its commit message says "Row groups inside a file are still read in file order".

What has not been checked:

  • whether the reader re-checks the TopK threshold at each row-group boundary, so it can skip row groups whose statistics can no longer contribute, and
  • whether it stops reading a file early once the threshold rules out the rest of it.

If neither happens, a TopK query still decodes every row group of each file it opens, even when the first one already fills the limit.

To answer this: run a TopK query over one file with several sorted row groups and count the row groups read.

Related: #66. A scan that stops early cancels in-flight inserts, which is what strands eviction victims there. Making TopK stop earlier will exercise that path more often.

Activity

  1. added
    questionFurther information is requested
    performanceWasted work, over-eviction or excess IO
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceWasted work, over-eviction or excess IOquestionFurther information is requested

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions