Skip to content

[python] Filter shard reads and _ROW_ID reads after selecting rows - #11024

Merged
JingsongLi merged 1 commit into
apache:masterfrom
LuciferYang:fix/py-shard-filter-positions
Nov 10, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
LuciferYang:fix/py-shard-filter-positions

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

close #11023

With a filter, with_shard and with_slice return rows that belong to another shard, and a filtered read of _ROW_ID on a row-tracking table returns renumbered _ROW_IDs. The issue has the repro. Both count the rows the format reader returns, and the format reader has already applied the pushed-down filter.

file_reader_supplier no longer pushes the filter into the format reader when it reads a shard range or the read assigns _ROW_ID, as it already does for deletion vectors and physical row ranges. RawFileSplitRead applies the filter to those reads after the rows are selected. Reads without a shard and without _ROW_ID still push the filter down.

#10723 changes the same condition in create_reader for Vortex files; it does not touch shard or _ROW_ID reads.

Tests

read_predicate_positions_test.py, all failing on master:

  • For parquet, orc and avro, shard 0 and shard 1 of a two-shard scan with an IN filter each return only their own row, with_slice(0, 5) returns only the row inside the slice, and a filtered read of _ROW_ID on a row-tracking table returns the rows' own ids.
  • For vortex, a sharded filtered read of a row-tracking table returns the right rows and _ROW_IDs in each shard.

@Akash3121 Akash3121 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 6aeb4ee into apache:master Nov 10, 2026
23 checks passed
@LuciferYang

Copy link
Copy Markdown
Contributor Author

Thank you @JingsongLi and @Akash3121

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] [Python] Filtered shard and _ROW_ID reads count rows after the format reader's filter

3 participants