Skip to content

duckdb: MultiFileReader interface - #9383

Merged
myrrc merged 37 commits into
developfrom
myrrc/duckdb-multi-file-reader
Aug 21, 2026
Merged

myrrc merged 37 commits into
developfrom
myrrc/duckdb-multi-file-reader

Conversation

@myrrc

@myrrc myrrc commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

Migrate vortex-duckdb to use MultiFileReader interface. We need this for four
reasons:

  1. Part of our current logic (virtual columns emission and pruning, partition
    stats, file pruning) is repeated in Vortex, but duckdb has this code written
    already, so it makes sense to reuse it.
  2. Duckdb 2.0 adds async IO interface via a separate prefetch thread pool which
    is available only with MultiFileReader.
  3. We want DuckLake integration Epic: Vortex DuckLake - maybe? #7820 which again is implemented using
    MultiFileReader.
  4. We want to support hive partitioning
    Hive partitioning in DuckDB #7746 but implementing it
    ourselves would be tedious.

In addition to MultiFileReader-related changes, this patch:

  • Adds a temporary upstream code patch with FinalizeScan call to
    MultiFileReaderInterface. We need it to support aggregate scans finalization.
    Will be removed once Push down ungrouped aggregates into file readers duckdb/duckdb#24697 is merged
    which bundles these changes, and
  • Fixes a bug in aggregate pushdown optimizer pass which changed new columns'
    names from rewritten column ids and not from original ids.
  • Fixes a memory leak of min/max values in GetStatistics passed for a non-
    string column.
  • Addes a test of multiple same accumulators for same column, namely
    SELECT min(x), min(x).
  • Builds vortex-duckdb code in release with NDEBUG so that D_ASSERTs are
    discarded.

While migrating to MultiFile reader, we removed handling of:

  • Emission of file_index column.
  • Filename and file_index-based pruning.
  • Column projections,
  • Statistics accumulation, and
  • Partition data/partition info callbacks.

We handle cardinality() callback differently now. Before the estimation was in
Vortex only based on all files, now we estimate cardinality based on first file
count which may be less accurate compared to the previous solution.

This change alters processing granularity from file- (partition-) based to
split-based. As a result, we no longer need async tasks, channels and
synchronisation above. Each thread gets its own split of a file (or next file
split if current file is exhausted) and blocks on it independently.

Resolves: #9355

@myrrc myrrc added changelog/performance A performance improvement action/bench-sql Run SQL benchmarks without Vortex Compact on this PR labels Aug 12, 2026
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Aug 12, 2026
@myrrc myrrc changed the title MultiFileReaderInterface for vortex-duckdb MultiFileReader interface for vortex-duckdb Aug 12, 2026
@myrrc myrrc added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Aug 12, 2026
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Aug 12, 2026
@myrrc
myrrc force-pushed the myrrc/duckdb-multi-file-reader branch from e3a7145 to de036bf Compare August 17, 2026 18:21
@myrrc myrrc added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Aug 18, 2026
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Aug 18, 2026
@myrrc
myrrc force-pushed the myrrc/duckdb-multi-file-reader branch from 3669f12 to 4c183cb Compare August 18, 2026 14:18
@myrrc myrrc changed the title MultiFileReader interface for vortex-duckdb duckdb: MultiFileReader interface Aug 18, 2026
@myrrc
myrrc requested a review from 0ax1 August 18, 2026 16:32
@myrrc
myrrc marked this pull request as ready for review August 18, 2026 16:41
@myrrc
myrrc requested review from gatesn and joseph-isaacs August 18, 2026 16:53
@joseph-isaacs joseph-isaacs added the action/bench-all Runs all CPU benchmarks. Do not combine with other benchmark labels; GPU is the only exception. label Aug 19, 2026
@github-actions github-actions Bot removed the action/bench-all Runs all CPU benchmarks. Do not combine with other benchmark labels; GPU is the only exception. label Aug 19, 2026
Comment thread vortex-duckdb/cpp/multi_file_reader.cpp Resolved
Comment thread vortex-duckdb/src/file_reader.rs Resolved
@myrrc
myrrc enabled auto-merge (squash) August 19, 2026 12:36
@myrrc
myrrc requested a review from joseph-isaacs August 19, 2026 12:40

@0ax1 0ax1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

couple of thoughts

Comment thread vortex-duckdb/cpp/multi_file_reader.cpp Resolved
Comment thread vortex-duckdb/cpp/multi_file_reader.cpp Resolved
Comment thread vortex-duckdb/build.rs Resolved
@myrrc
myrrc force-pushed the myrrc/duckdb-multi-file-reader branch from 8296fc4 to c6c5f0e Compare August 20, 2026 09:51
@myrrc
myrrc requested a review from 0ax1 August 20, 2026 09:51
@myrrc
myrrc disabled auto-merge August 20, 2026 10:15
@joseph-isaacs joseph-isaacs added the action/bench-all Runs all CPU benchmarks. Do not combine with other benchmark labels; GPU is the only exception. label Aug 20, 2026
@github-actions github-actions Bot removed the action/bench-all Runs all CPU benchmarks. Do not combine with other benchmark labels; GPU is the only exception. label Aug 20, 2026
myrrc added 16 commits August 21, 2026 11:55
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
@myrrc
myrrc force-pushed the myrrc/duckdb-multi-file-reader branch from 56acded to 1c0733e Compare August 21, 2026 11:11
@myrrc
myrrc requested a review from joseph-isaacs August 21, 2026 11:11
@joseph-isaacs joseph-isaacs added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Aug 21, 2026
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Aug 21, 2026
@myrrc
myrrc removed request for 0ax1 and gatesn August 21, 2026 14:58

@joseph-isaacs joseph-isaacs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We accept the 2x regression in fineweb:duckdb:vortex q3

We know how to fix this going fowards

@myrrc
myrrc merged commit acd8bb4 into develop Aug 21, 2026
115 of 116 checks passed
@myrrc
myrrc deleted the myrrc/duckdb-multi-file-reader branch August 21, 2026 16:38
@myrrc

myrrc commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

The fineweb 3 regression is due to us doing more work per split:

15.5x base onpair::decoding::try_decode_into::<onpair::core::dictionary::compact::CompactDictionaryView>
16x base <u16 as fastlanes::bitpacking::BitPacking>::unpack::<12, 768>

Will be addressed shortly

myrrc added a commit that referenced this pull request Sep 16, 2026
#9383 removed handling of
projection_ids, since MultiFileReader handles them internally.

However, if there's a table filter in get.table_filters which is passed
to
Vortex and this filter contains a column that's not part of projection,
this
column will stay in column_ids, and Vortex will load and canonicalize it
which
is extra work.

Main reason for this change is the removal for the "Eq not pushed" hack
up the stack - if we save some filters in table filters, we don't want
to project these filter-only columns still in column_ids.

---------

Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MultiFileReader interface for vortex-duckdb

3 participants