Skip to content

Run DataFusion's Scan API tables on the V2 scan - #11626

Closed
gatesn wants to merge 26 commits into
ngates/v2-scan-poolfrom
ngates/datasource-v2-plans-2d0543
Closed

gatesn wants to merge 26 commits into
ngates/v2-scan-poolfrom
ngates/datasource-v2-plans-2d0543

Conversation

@gatesn

@gatesn gatesn commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

The DataFusion Scan API table path in datafusion-bench (VortexTable over a MultiFileDataSource) only ran with the V1 scan, because it always called MultiFileDataSource::build, which opens a LayoutReader per file. #11665 gives the multi-file source a V2 form, build_plan, a PlanDataSource whose files open as the scan reaches them. With VORTEX_USE_SCAN_API=1, the table path now picks between the two with --scan-v1, so the Scan API benchmark row can measure VortexTable on V2.

This replaces the earlier version of this PR. That version added a UnionDataSource over per-file sources and row-range partitions. PlanDataSource::from_children in #11665 now covers the multi-file union natively, and row-range partitions were dropped from the stack.

Behaviour change: VORTEX_USE_SCAN_API=1 together with --scan-v1 false used to register a listing table and ignore the variable. It now registers the Scan API table on V2.

Stacked on #11665. #11685 replaces the environment variable with a --scan-api flag; whichever lands second takes the other's change.

🤖 Generated with Claude Code

gatesn and others added 4 commits November 18, 2026 20:20
Move SegmentId, SegmentSource, SegmentLocation, the coalescing
FileSegmentSource and its read driver, BufferSegmentSource and
SharedSegmentSource into vortex_io::segments, so the I/O layer can read
segments without depending on vortex-file or vortex-layout. The old
paths re-export the moved items; behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Scan I/O requests carry a SourceId and a segment target instead of a byte
range, and one IoScheduler per session serves them. Placed sources read
through the coalescing FileSegmentSource, serve the segments the footer
read already fetched, share in-flight reads through SharedSegmentSource,
and reuse a live registration across concurrent scans of one file; opaque
sources read segments themselves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
A partition can now return its scan as a `PartitionPlan::Scan` builder from
`Partition::plan`, so engines that drive the V2 executor themselves reach it
through `DataSource` instead of a concrete `VortexFile`. `PlannedDataSource`
is the standard implementation for any source that can name its segments, and
`VortexFile::planned_data_source` exposes a file through it. The DataFusion
opener and DuckDB's V2 scan now plan through the file's data source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
The opener bound and optimized the projection and filter to learn the scan
dtype, and the data source bound them again. A V2 scan now takes its dtype
from the data source's scan, and only V1 binds in the opener. A partition
with one stream returns it without a select wrapper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn added the changelog/feature A new feature label Nov 19, 2026
@gatesn

gatesn commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

TPC-H SF1, DataFusion --scan-v1 false, release_debug, local M-series. Four rounds × 10 iterations, alternating base (a11813a), branch, and a byte-identical copy of base (A/A noise control). Milliseconds are medians over all iterations. The paired ratio is the median of per-round ratios.

Branch vs base

query base ms branch ms ratio paired ratio (median of rounds)
tpch_q01 23.6 23.4 0.990 0.994
tpch_q02 9.7 9.6 0.996 0.996
tpch_q03 11.3 11.5 1.016 1.005
tpch_q04 8.2 8.1 0.986 1.005
tpch_q05 20.4 19.2 0.941 0.990
tpch_q06 3.3 3.2 0.958 0.962
tpch_q07 14.7 14.1 0.958 1.000
tpch_q08 15.6 15.2 0.974 0.988
tpch_q09 21.2 21.2 1.000 0.997
tpch_q10 13.9 13.1 0.940 1.010
tpch_q11 6.6 6.3 0.954 0.967
tpch_q12 8.2 8.2 1.001 0.982
tpch_q13 9.6 9.4 0.978 1.021
tpch_q14 5.3 5.1 0.961 0.952
tpch_q15 8.6 8.3 0.973 0.967
tpch_q16 8.1 7.6 0.932 0.972
tpch_q17 32.8 32.1 0.977 0.990
tpch_q18 35.3 33.9 0.960 1.003
tpch_q19 7.6 7.2 0.945 0.990
tpch_q20 12.6 11.4 0.907 0.978
tpch_q21 25.6 23.2 0.905 0.995
tpch_q22 5.4 4.8 0.886 0.960
sum 307.6 295.9 0.962

geomean ratio 0.960; geomean paired 0.987; 4 rounds x 10 iterations

A/A: base copy vs base

query base ms base copy ms ratio paired ratio (median of rounds)
tpch_q01 23.6 23.9 1.012 1.029
tpch_q02 9.7 9.9 1.024 1.006
tpch_q03 11.3 11.6 1.029 1.005
tpch_q04 8.2 8.1 0.981 1.007
tpch_q05 20.4 19.7 0.966 1.017
tpch_q06 3.3 3.3 0.986 0.972
tpch_q07 14.7 14.4 0.980 1.001
tpch_q08 15.6 17.2 1.107 1.009
tpch_q09 21.2 21.9 1.032 1.006
tpch_q10 13.9 13.3 0.951 1.019
tpch_q11 6.6 6.2 0.938 0.978
tpch_q12 8.2 8.1 0.992 0.999
tpch_q13 9.6 9.8 1.014 0.986
tpch_q14 5.3 5.0 0.944 0.956
tpch_q15 8.6 8.5 0.988 0.989
tpch_q16 8.1 7.4 0.913 0.987
tpch_q17 32.8 31.7 0.965 1.004
tpch_q18 35.3 34.6 0.981 0.996
tpch_q19 7.6 6.9 0.916 0.973
tpch_q20 12.6 11.1 0.881 0.985
tpch_q21 25.6 24.0 0.938 0.967
tpch_q22 5.4 4.9 0.915 0.966
sum 307.6 301.5 0.980

geomean ratio 0.974; geomean paired 0.993; 4 rounds x 10 iterations

gatesn and others added 4 commits November 18, 2026 22:10
Engines will take a scan split by split through the Scan API, so a hook that
hands them the scan builder would be removed again. The DataFusion file opener
and DuckDB go back to driving the V2 executor themselves; the planned data
source and its partitions stay, running the scan from `execute`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
VortexFile::data_source() now returns the planner-driven source, with
file-statistics pruning in scan(), and falls back to the layout reader
for files whose layout tree has a list layout, which V2 cannot read yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
MultiFileDataSource now builds a format-agnostic vortex_scan::multi::MultiDataSource
over each file's data_source(), keeping lazy opening, per-file pruning,
partition selection by file ordinal and the shared unordered limit.
datafusion-bench gains --scan-api to register tables through VortexTable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
An unordered, unlimited scan of a PlannedDataSource now yields one
partition per row range, up to the available parallelism and no smaller
than 65,536 rows, so engines that parallelise by partition spread a
single file across their workers. Through DataFusion's VortexTable, a
single-file table otherwise ran on one output partition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn changed the title Let any data source hand engines a planner-driven scan Run Scan API sources on the V2 scan Nov 19, 2026
@gatesn gatesn added changelog/break A breaking API change and removed changelog/feature A new feature labels Nov 19, 2026
@gatesn

gatesn commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Same datafusion-bench binary (release_debug), local, sequential rounds alternating ABBA. base = file opener V2 (--scan-v1 false), branch = VortexTable over MultiFileDataSource on V2 (--scan-v1 false --scan-api true), A/A = a byte-identical copy of the binary on the opener path.

TPC-H SF1 (4 rounds × 10 iterations)

query base ms branch ms ratio paired ratio (median of rounds)
tpch_q01 21.6 26.3 1.214 1.220
tpch_q02 8.1 7.0 0.860 0.863
tpch_q03 10.9 7.3 0.674 0.668
tpch_q04 7.3 6.3 0.854 0.859
tpch_q05 18.2 16.2 0.891 0.896
tpch_q06 2.9 1.5 0.525 0.517
tpch_q07 12.9 52.1 4.047 4.084
tpch_q08 14.2 12.5 0.878 0.887
tpch_q09 20.0 23.4 1.172 1.167
tpch_q10 12.2 16.6 1.356 1.362
tpch_q11 5.6 4.3 0.779 0.776
tpch_q12 7.9 5.9 0.747 0.742
tpch_q13 10.1 12.2 1.213 1.226
tpch_q14 4.7 3.4 0.733 0.727
tpch_q15 7.8 5.0 0.649 0.650
tpch_q16 7.2 6.8 0.938 0.937
tpch_q17 30.5 31.8 1.045 1.052
tpch_q18 34.0 53.0 1.559 1.562
tpch_q19 5.8 3.8 0.650 0.644
tpch_q20 10.2 9.0 0.884 0.898
tpch_q21 21.0 48.8 2.328 2.331
tpch_q22 4.3 7.0 1.623 1.622
sum 277.3 360.3 1.299

geomean ratio 1.026; geomean paired 1.027; 4 rounds x 10 iterations

A/A: geomean ratio 0.997; geomean paired 1.002; 4 rounds x 10 iterations

TPC-DS SF1 (3 rounds × 5 iterations, q72 excluded: ~95 GB RSS through VortexTable on base V1 as well)

98 queries
query base ms branch ms ratio paired ratio (median of rounds)
tpcds_q01 7.7 7.7 0.991 0.956
tpcds_q02 11.5 12.9 1.120 1.120
tpcds_q03 4.8 5.7 1.204 1.196
tpcds_q04 62.3 81.0 1.300 1.283
tpcds_q05 12.7 10.2 0.804 0.785
tpcds_q06 7.3 6.6 0.898 0.898
tpcds_q07 10.0 8.8 0.881 0.861
tpcds_q08 8.5 8.2 0.962 0.943
tpcds_q09 10.8 8.2 0.755 0.746
tpcds_q10 13.7 12.0 0.878 0.852
tpcds_q11 32.4 47.7 1.475 1.467
tpcds_q12 4.3 6.0 1.410 1.375
tpcds_q13 11.2 9.6 0.854 0.853
tpcds_q14 53.9 162.4 3.013 3.008
tpcds_q15 4.8 8.7 1.817 1.785
tpcds_q16 7.0 5.4 0.766 0.754
tpcds_q17 13.4 10.1 0.754 0.744
tpcds_q18 15.1 15.7 1.039 1.025
tpcds_q19 7.8 6.6 0.852 0.847
tpcds_q20 4.2 7.8 1.876 1.878
tpcds_q21 5.7 71.8 12.658 12.426
tpcds_q22 74.3 214.4 2.886 2.886
tpcds_q23 59.3 122.8 2.071 2.107
tpcds_q24 21.4 23.5 1.099 1.092
tpcds_q25 12.7 9.6 0.754 0.749
tpcds_q26 8.0 7.5 0.945 0.944
tpcds_q27 21.0 16.2 0.773 0.788
tpcds_q28 11.1 6.7 0.603 0.595
tpcds_q29 13.9 10.4 0.748 0.745
tpcds_q30 8.7 10.6 1.224 1.240
tpcds_q31 23.5 17.8 0.758 0.761
tpcds_q32 4.5 3.0 0.676 0.667
tpcds_q33 9.4 6.9 0.737 0.724
tpcds_q34 8.4 9.2 1.098 1.121
tpcds_q35 15.4 14.1 0.913 0.903
tpcds_q36 17.8 16.8 0.943 0.925
tpcds_q37 3.7 4.4 1.183 1.160
tpcds_q38 12.3 27.6 2.245 2.275
tpcds_q39 23.0 123.8 5.387 5.393
tpcds_q40 5.4 16.3 3.000 2.950
tpcds_q41 5.2 3.2 0.610 0.615
tpcds_q42 4.6 3.9 0.863 0.880
tpcds_q43 6.4 5.6 0.876 0.874
tpcds_q44 9.2 7.2 0.786 0.780
tpcds_q45 5.4 9.7 1.795 1.791
tpcds_q46 11.6 18.6 1.606 1.572
tpcds_q47 44.8 40.7 0.909 0.899
tpcds_q48 9.8 11.0 1.128 1.094
tpcds_q49 16.8 12.8 0.760 0.745
tpcds_q50 12.6 22.0 1.742 1.722
tpcds_q51 37.8 36.3 0.960 0.970
tpcds_q52 5.0 4.2 0.828 0.855
tpcds_q53 6.8 7.6 1.115 1.129
tpcds_q54 9.4 9.3 0.988 1.045
tpcds_q55 4.7 4.1 0.868 0.884
tpcds_q56 9.6 7.1 0.739 0.717
tpcds_q57 32.7 26.9 0.824 0.831
tpcds_q58 12.6 32.0 2.533 2.540
tpcds_q59 13.7 19.0 1.386 1.386
tpcds_q60 9.7 7.7 0.798 0.798
tpcds_q61 11.7 71.3 6.092 6.075
tpcds_q62 5.4 104.5 19.233 43.657
tpcds_q63 6.4 7.2 1.115 1.116
tpcds_q64 61.5 60.4 0.983 0.978
tpcds_q65 24.8 11.5 0.462 0.460
tpcds_q66 18.4 31.3 1.703 1.710
tpcds_q67 59.5 191.1 3.210 3.217
tpcds_q68 10.8 9.9 0.916 0.903
tpcds_q69 14.1 11.0 0.775 0.783
tpcds_q70 35.2 49.2 1.397 1.418
tpcds_q71 7.9 6.4 0.805 0.797
tpcds_q73 8.1 6.2 0.763 0.757
tpcds_q74 22.2 27.4 1.233 1.232
tpcds_q75 31.5 37.4 1.187 1.199
tpcds_q76 7.1 8.9 1.250 1.246
tpcds_q77 12.4 9.3 0.752 0.750
tpcds_q78 27.2 22.7 0.836 0.840
tpcds_q79 10.5 12.9 1.223 1.230
tpcds_q80 17.8 13.7 0.769 0.771
tpcds_q81 9.2 9.6 1.042 1.052
tpcds_q82 4.5 5.0 1.093 1.079
tpcds_q83 11.0 13.0 1.184 1.170
tpcds_q84 4.4 4.7 1.069 1.069
tpcds_q85 14.7 10.3 0.700 0.698
tpcds_q86 4.2 4.2 1.000 1.000
tpcds_q87 12.2 27.3 2.241 2.245
tpcds_q88 18.7 16.8 0.899 0.892
tpcds_q89 7.6 10.8 1.416 1.435
tpcds_q90 4.3 3.1 0.734 0.726
tpcds_q91 6.0 5.7 0.948 0.937
tpcds_q92 4.7 3.2 0.685 0.694
tpcds_q93 10.5 9.5 0.913 0.916
tpcds_q94 6.4 4.7 0.730 0.739
tpcds_q95 16.4 12.1 0.741 0.740
tpcds_q96 4.2 3.5 0.844 0.837
tpcds_q97 10.3 10.3 1.007 0.983
tpcds_q98 5.6 14.7 2.618 2.580
tpcds_q99 8.3 968.1 116.514 118.185
sum 1487.0 3292.5 2.214

geomean ratio 1.214; geomean paired 1.220; 3 rounds x 5 iterations

A/A: geomean ratio 0.990; geomean paired 0.990; 3 rounds x 5 iterations

The largest TPC-DS gaps (q99, q62) use the same join modes but a different build side: VortexTable reports no column statistics, so DataFusion builds the hash table on catalog_sales rather than the dimension. On base, the VortexTable path on V1 took 3065 ms for q99, against 1000 ms here and 8 ms for the opener.

@robert3005

Copy link
Copy Markdown
Contributor

Any idea what's the cause of the worst regressions in this pr?

@gatesn

gatesn commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

These numbers compare two engine integrations, not before and after this PR. On base, the VortexTable (Scan API) path ran on V1, and this PR makes it faster everywhere I looked. On a single-iteration smoke run, TPC-H took about 1200 ms on base and about 360 ms here. TPC-DS q99 went from 3065 ms to 1000 ms. The table measures what's still left between VortexTable and the DataFusion file opener, now that both run the V2 scan.

The worst gaps I looked at come from the integration, not the scan:

  • Join build side. VortexTable exposes only table-level row and byte counts, with no column statistics. DataFusion then picks the fact table as the hash-join build side. That happens in TPC-DS q99 and q62, where the join modes are identical but catalog_sales is the side that gets collected. It also accounts for TPC-H q7 and q21, which become CollectLeft instead of Partitioned, and TPC-H q18, where the semi-join sides are swapped. TPC-DS q72 needs about 95 GB this way, on base as well.
  • Projection pushdown. The opener pushes computed projections such as q1's l_extendedprice * (1 - l_discount) into the Vortex scan, and VortexTable evaluates them in DataFusion.
  • Ruled out. I capped each partition's in-flight work by the number of splits, and it made no difference. The results also match the opener on every TPC-H and TPC-DS query.

I haven't looked at TPC-H q10, q13 or q22 yet. The fixes for the two gaps above belong in vortex-datafusion's VortexTable: column statistics and projection pushdown. I've kept them out of this PR.

gatesn and others added 7 commits November 19, 2026 12:26
PlannedDataSource becomes PlanDataSource and MultiDataSource becomes
UnionDataSource (module vortex_scan::union), with DataSourceFactory now
DeferredDataSource. ScanRequest gains target_partitions, which replaces
PlanDataSource::with_max_partitions: engines that know their parallelism
(DataFusion's target partitions) pass it, and a union divides it across the
children it scans. Because the split count now depends on the request, a
PlanDataSource is one selectable unit with index 0, and the Scan API docs say
what an index names and how row_range and selection apply to each unit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Benchmarking VortexTable needed both --scan-v1 true and
VORTEX_USE_SCAN_API=1, so it could not be compared with the listing table's
V2 scan in one run. --scan-api true registers Vortex tables through the Scan
API whatever --scan-v1 says; the environment variable keeps working.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Engines that consume the Scan API still scanned Vortex files with the V1
layout reader. PlanDataSource exposes any ScanInput as a DataSource whose
single partition runs on the planner-driven ScanDriver, and
VortexFile::data_source now uses it, falling back to the layout reader for
files with list layouts, which the V2 executor cannot read yet. As the
layout reader did, a planned file scan whose filter the footer statistics
prove false reads nothing.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A PlanDataSource scan had one partition, so engines that parallelise by
partition read a whole file on one worker. Unordered, unlimited scans now
split the row range into near-equal partitions, sized by the new
ScanRequest::target_partitions hint or the available parallelism, with at
most one partition per started 65,536 rows. DataFusion passes its target
partition count.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A split does not keep every partition above 65,536 rows: it makes at most one
partition per started 65,536 rows, so 100,000 rows split into two halves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Records the split stack as this branch's base without changing its tree, so the
pull request shows only the union layer.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scan I/O requests carry a SourceId and a segment target instead of a byte
range, and one IoScheduler per session serves them. Placed sources read
through the coalescing FileSegmentSource, serve the segments the footer
read already fetched, share in-flight reads through SharedSegmentSource,
and reuse a live registration across concurrent scans of one file; opaque
sources read segments themselves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn force-pushed the ngates/scan-io-segment-requests branch from a11813a to 2a14cbb Compare November 19, 2026 18:23
gatesn and others added 5 commits November 19, 2026 13:25
Benchmarking VortexTable needed both --scan-v1 true and
VORTEX_USE_SCAN_API=1, so it could not be compared with the listing table's
V2 scan in one run. --scan-api true registers Vortex tables through the Scan
API whatever --scan-v1 says; the environment variable keeps working.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Engines that consume the Scan API still scanned Vortex files with the V1
layout reader. PlanDataSource exposes any ScanInput as a DataSource whose
single partition runs on the planner-driven ScanDriver, and
VortexFile::data_source now uses it, falling back to the layout reader for
files with list layouts, which the V2 executor cannot read yet. As the
layout reader did, a planned file scan whose filter the footer statistics
prove false reads nothing.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A PlanDataSource scan had one partition, so engines that parallelise by
partition read a whole file on one worker. Unordered, unlimited scans now
split the row range into near-equal partitions, sized by the new
ScanRequest::target_partitions hint or the available parallelism, with at
most one partition per started 65,536 rows. DataFusion passes its target
partition count.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Records the split stack as this branch's base without changing its tree, so the
pull request shows only the union layer once it targets that branch.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gatesn gatesn changed the title Run Scan API sources on the V2 scan Run multi-file Scan API sources on the V2 scan Nov 19, 2026
@gatesn
gatesn changed the base branch from ngates/scan-io-segment-requests to ngates/scan-request-target-partitions November 19, 2026 18:32
gatesn and others added 3 commits November 19, 2026 14:56
Engines that keep a V1/V2 switch while they move onto the Scan API, and
benchmarks comparing the two executors, need each one as a DataSource.
layout_reader_data_source() returns the layout-reader source that
data_source() already falls back to, with the same footer statistics, and
data_source() now calls it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gatesn and others added 2 commits November 20, 2026 11:42
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
#11626's union layer is superseded by PlanDataSource::from_children on
#11665, which MultiFileDataSource::build_plan already builds. This merge
keeps the branch's history and takes #11665 plus the table path change
as its tree.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gatesn
gatesn changed the base branch from ngates/scan-request-target-partitions to ngates/v2-scan-pool November 20, 2026 16:42
@gatesn gatesn changed the title Run multi-file Scan API sources on the V2 scan Run DataFusion's Scan API tables on the V2 scan Nov 20, 2026
@gatesn gatesn added changelog/chore A trivial change and removed changelog/break A breaking API change labels Nov 20, 2026
gatesn added a commit that referenced this pull request Nov 20, 2026
A Vortex benchmark target now names both axes: the integration (`file`
through the engine's file format, `datasource` through DataFusion's
`VortexTable`) and the scan engine (`v1` LayoutReader, `v2` plans).
The scan engine was implicit per suite; suites that cannot run v2 yet
are now only a constraint, not the selector.

Results carry both axes as `integration` and `scan_engine` fields, and
every combination but the file integration on the suite's default scan
appends `[integration, scan]` to its format identity, so existing
history continues. Validation rejects datasource-v2 (needs #11626) and
DuckDB datasource (no table function yet).

PR and develop run every valid combination on ClickBench and TPC-H SF1
NVMe; pr-full runs them on every suite, and nightly runs the other
scans in companion jobs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
gatesn added a commit that referenced this pull request Nov 20, 2026
With `--scan-api true --scan-v1 false`, datafusion-bench registers a `VortexTable` over `MultiFileDataSource::build_plan`, so the datasource integration can run the V2 scan. The orchestrator now accepts datafusion datasource-v2 targets, and every-scan coverage runs them. Folds in #11626.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn

gatesn commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Folded into #11665 at bdc7a33: datafusion-bench's Scan API tables run on MultiFileDataSource::build_plan under --scan-api true --scan-v1 false, and the orchestrator now accepts DataFusion datasource-v2 targets on #11685's integration × scan-engine axes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants