Repository navigation
duckdb: MultiFileReader interface - #9383
Merged
Merged
Conversation
myrrc
force-pushed
the
myrrc/duckdb-multi-file-reader
branch
from
August 17, 2026 18:21
e3a7145 to
de036bf
Compare
myrrc
force-pushed
the
myrrc/duckdb-multi-file-reader
branch
from
August 18, 2026 14:18
3669f12 to
4c183cb
Compare
myrrc
marked this pull request as ready for review
August 18, 2026 16:41
myrrc
enabled auto-merge (squash)
August 19, 2026 12:36
0ax1
reviewed
Aug 19, 2026
myrrc
force-pushed
the
myrrc/duckdb-multi-file-reader
branch
from
August 20, 2026 09:51
8296fc4 to
c6c5f0e
Compare
myrrc
disabled auto-merge
August 20, 2026 10:15
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
myrrc
force-pushed
the
myrrc/duckdb-multi-file-reader
branch
from
August 21, 2026 11:11
56acded to
1c0733e
Compare
joseph-isaacs
approved these changes
Aug 21, 2026
joseph-isaacs
left a comment
Contributor
There was a problem hiding this comment.
We accept the 2x regression in fineweb:duckdb:vortex q3
We know how to fix this going fowards
Contributor
Author
|
The fineweb 3 regression is due to us doing more work per split: Will be addressed shortly |
myrrc
added a commit
that referenced
this pull request
Sep 16, 2026
#9383 removed handling of projection_ids, since MultiFileReader handles them internally. However, if there's a table filter in get.table_filters which is passed to Vortex and this filter contains a column that's not part of projection, this column will stay in column_ids, and Vortex will load and canonicalize it which is extra work. Main reason for this change is the removal for the "Eq not pushed" hack up the stack - if we save some filters in table filters, we don't want to project these filter-only columns still in column_ids. --------- Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrate vortex-duckdb to use MultiFileReader interface. We need this for four
reasons:
stats, file pruning) is repeated in Vortex, but duckdb has this code written
already, so it makes sense to reuse it.
is available only with MultiFileReader.
MultiFileReader.
Hive partitioning in DuckDB #7746 but implementing it
ourselves would be tedious.
In addition to MultiFileReader-related changes, this patch:
MultiFileReaderInterface. We need it to support aggregate scans finalization.
Will be removed once Push down ungrouped aggregates into file readers duckdb/duckdb#24697 is merged
which bundles these changes, and
names from rewritten column ids and not from original ids.
string column.
SELECT min(x), min(x).NDEBUGso thatD_ASSERTs arediscarded.
While migrating to MultiFile reader, we removed handling of:
file_indexcolumn.file_index-based pruning.We handle
cardinality()callback differently now. Before the estimation was inVortex only based on all files, now we estimate cardinality based on first file
count which may be less accurate compared to the previous solution.
This change alters processing granularity from file- (partition-) based to
split-based. As a result, we no longer need async tasks, channels and
synchronisation above. Each thread gets its own split of a file (or next file
split if current file is exhausted) and blocks on it independently.
Resolves: #9355