Repository navigation
Conversation
Move SegmentId, SegmentSource, SegmentLocation, the coalescing FileSegmentSource and its read driver, BufferSegmentSource and SharedSegmentSource into vortex_io::segments, so the I/O layer can read segments without depending on vortex-file or vortex-layout. The old paths re-export the moved items; behaviour is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Scan I/O requests carry a SourceId and a segment target instead of a byte range, and one IoScheduler per session serves them. Placed sources read through the coalescing FileSegmentSource, serve the segments the footer read already fetched, share in-flight reads through SharedSegmentSource, and reuse a live registration across concurrent scans of one file; opaque sources read segments themselves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
A partition can now return its scan as a `PartitionPlan::Scan` builder from `Partition::plan`, so engines that drive the V2 executor themselves reach it through `DataSource` instead of a concrete `VortexFile`. `PlannedDataSource` is the standard implementation for any source that can name its segments, and `VortexFile::planned_data_source` exposes a file through it. The DataFusion opener and DuckDB's V2 scan now plan through the file's data source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
The opener bound and optimized the projection and filter to learn the scan dtype, and the data source bound them again. A V2 scan now takes its dtype from the data source's scan, and only V1 binds in the opener. A partition with one stream returns it without a select wrapper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
|
TPC-H SF1, DataFusion Branch vs base
geomean ratio 0.960; geomean paired 0.987; 4 rounds x 10 iterations A/A: base copy vs base
geomean ratio 0.974; geomean paired 0.993; 4 rounds x 10 iterations |
Engines will take a scan split by split through the Scan API, so a hook that hands them the scan builder would be removed again. The DataFusion file opener and DuckDB go back to driving the V2 executor themselves; the planned data source and its partitions stay, running the scan from `execute`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
VortexFile::data_source() now returns the planner-driven source, with file-statistics pruning in scan(), and falls back to the layout reader for files whose layout tree has a list layout, which V2 cannot read yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
MultiFileDataSource now builds a format-agnostic vortex_scan::multi::MultiDataSource over each file's data_source(), keeping lazy opening, per-file pruning, partition selection by file ordinal and the shared unordered limit. datafusion-bench gains --scan-api to register tables through VortexTable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
An unordered, unlimited scan of a PlannedDataSource now yields one partition per row range, up to the available parallelism and no smaller than 65,536 rows, so engines that parallelise by partition spread a single file across their workers. Through DataFusion's VortexTable, a single-file table otherwise ran on one output partition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
|
Same TPC-H SF1 (4 rounds × 10 iterations)
geomean ratio 1.026; geomean paired 1.027; 4 rounds x 10 iterations A/A: geomean ratio 0.997; geomean paired 1.002; 4 rounds x 10 iterations TPC-DS SF1 (3 rounds × 5 iterations, q72 excluded: ~95 GB RSS through
|
| query | base ms | branch ms | ratio | paired ratio (median of rounds) |
|---|---|---|---|---|
| tpcds_q01 | 7.7 | 7.7 | 0.991 | 0.956 |
| tpcds_q02 | 11.5 | 12.9 | 1.120 | 1.120 |
| tpcds_q03 | 4.8 | 5.7 | 1.204 | 1.196 |
| tpcds_q04 | 62.3 | 81.0 | 1.300 | 1.283 |
| tpcds_q05 | 12.7 | 10.2 | 0.804 | 0.785 |
| tpcds_q06 | 7.3 | 6.6 | 0.898 | 0.898 |
| tpcds_q07 | 10.0 | 8.8 | 0.881 | 0.861 |
| tpcds_q08 | 8.5 | 8.2 | 0.962 | 0.943 |
| tpcds_q09 | 10.8 | 8.2 | 0.755 | 0.746 |
| tpcds_q10 | 13.7 | 12.0 | 0.878 | 0.852 |
| tpcds_q11 | 32.4 | 47.7 | 1.475 | 1.467 |
| tpcds_q12 | 4.3 | 6.0 | 1.410 | 1.375 |
| tpcds_q13 | 11.2 | 9.6 | 0.854 | 0.853 |
| tpcds_q14 | 53.9 | 162.4 | 3.013 | 3.008 |
| tpcds_q15 | 4.8 | 8.7 | 1.817 | 1.785 |
| tpcds_q16 | 7.0 | 5.4 | 0.766 | 0.754 |
| tpcds_q17 | 13.4 | 10.1 | 0.754 | 0.744 |
| tpcds_q18 | 15.1 | 15.7 | 1.039 | 1.025 |
| tpcds_q19 | 7.8 | 6.6 | 0.852 | 0.847 |
| tpcds_q20 | 4.2 | 7.8 | 1.876 | 1.878 |
| tpcds_q21 | 5.7 | 71.8 | 12.658 | 12.426 |
| tpcds_q22 | 74.3 | 214.4 | 2.886 | 2.886 |
| tpcds_q23 | 59.3 | 122.8 | 2.071 | 2.107 |
| tpcds_q24 | 21.4 | 23.5 | 1.099 | 1.092 |
| tpcds_q25 | 12.7 | 9.6 | 0.754 | 0.749 |
| tpcds_q26 | 8.0 | 7.5 | 0.945 | 0.944 |
| tpcds_q27 | 21.0 | 16.2 | 0.773 | 0.788 |
| tpcds_q28 | 11.1 | 6.7 | 0.603 | 0.595 |
| tpcds_q29 | 13.9 | 10.4 | 0.748 | 0.745 |
| tpcds_q30 | 8.7 | 10.6 | 1.224 | 1.240 |
| tpcds_q31 | 23.5 | 17.8 | 0.758 | 0.761 |
| tpcds_q32 | 4.5 | 3.0 | 0.676 | 0.667 |
| tpcds_q33 | 9.4 | 6.9 | 0.737 | 0.724 |
| tpcds_q34 | 8.4 | 9.2 | 1.098 | 1.121 |
| tpcds_q35 | 15.4 | 14.1 | 0.913 | 0.903 |
| tpcds_q36 | 17.8 | 16.8 | 0.943 | 0.925 |
| tpcds_q37 | 3.7 | 4.4 | 1.183 | 1.160 |
| tpcds_q38 | 12.3 | 27.6 | 2.245 | 2.275 |
| tpcds_q39 | 23.0 | 123.8 | 5.387 | 5.393 |
| tpcds_q40 | 5.4 | 16.3 | 3.000 | 2.950 |
| tpcds_q41 | 5.2 | 3.2 | 0.610 | 0.615 |
| tpcds_q42 | 4.6 | 3.9 | 0.863 | 0.880 |
| tpcds_q43 | 6.4 | 5.6 | 0.876 | 0.874 |
| tpcds_q44 | 9.2 | 7.2 | 0.786 | 0.780 |
| tpcds_q45 | 5.4 | 9.7 | 1.795 | 1.791 |
| tpcds_q46 | 11.6 | 18.6 | 1.606 | 1.572 |
| tpcds_q47 | 44.8 | 40.7 | 0.909 | 0.899 |
| tpcds_q48 | 9.8 | 11.0 | 1.128 | 1.094 |
| tpcds_q49 | 16.8 | 12.8 | 0.760 | 0.745 |
| tpcds_q50 | 12.6 | 22.0 | 1.742 | 1.722 |
| tpcds_q51 | 37.8 | 36.3 | 0.960 | 0.970 |
| tpcds_q52 | 5.0 | 4.2 | 0.828 | 0.855 |
| tpcds_q53 | 6.8 | 7.6 | 1.115 | 1.129 |
| tpcds_q54 | 9.4 | 9.3 | 0.988 | 1.045 |
| tpcds_q55 | 4.7 | 4.1 | 0.868 | 0.884 |
| tpcds_q56 | 9.6 | 7.1 | 0.739 | 0.717 |
| tpcds_q57 | 32.7 | 26.9 | 0.824 | 0.831 |
| tpcds_q58 | 12.6 | 32.0 | 2.533 | 2.540 |
| tpcds_q59 | 13.7 | 19.0 | 1.386 | 1.386 |
| tpcds_q60 | 9.7 | 7.7 | 0.798 | 0.798 |
| tpcds_q61 | 11.7 | 71.3 | 6.092 | 6.075 |
| tpcds_q62 | 5.4 | 104.5 | 19.233 | 43.657 |
| tpcds_q63 | 6.4 | 7.2 | 1.115 | 1.116 |
| tpcds_q64 | 61.5 | 60.4 | 0.983 | 0.978 |
| tpcds_q65 | 24.8 | 11.5 | 0.462 | 0.460 |
| tpcds_q66 | 18.4 | 31.3 | 1.703 | 1.710 |
| tpcds_q67 | 59.5 | 191.1 | 3.210 | 3.217 |
| tpcds_q68 | 10.8 | 9.9 | 0.916 | 0.903 |
| tpcds_q69 | 14.1 | 11.0 | 0.775 | 0.783 |
| tpcds_q70 | 35.2 | 49.2 | 1.397 | 1.418 |
| tpcds_q71 | 7.9 | 6.4 | 0.805 | 0.797 |
| tpcds_q73 | 8.1 | 6.2 | 0.763 | 0.757 |
| tpcds_q74 | 22.2 | 27.4 | 1.233 | 1.232 |
| tpcds_q75 | 31.5 | 37.4 | 1.187 | 1.199 |
| tpcds_q76 | 7.1 | 8.9 | 1.250 | 1.246 |
| tpcds_q77 | 12.4 | 9.3 | 0.752 | 0.750 |
| tpcds_q78 | 27.2 | 22.7 | 0.836 | 0.840 |
| tpcds_q79 | 10.5 | 12.9 | 1.223 | 1.230 |
| tpcds_q80 | 17.8 | 13.7 | 0.769 | 0.771 |
| tpcds_q81 | 9.2 | 9.6 | 1.042 | 1.052 |
| tpcds_q82 | 4.5 | 5.0 | 1.093 | 1.079 |
| tpcds_q83 | 11.0 | 13.0 | 1.184 | 1.170 |
| tpcds_q84 | 4.4 | 4.7 | 1.069 | 1.069 |
| tpcds_q85 | 14.7 | 10.3 | 0.700 | 0.698 |
| tpcds_q86 | 4.2 | 4.2 | 1.000 | 1.000 |
| tpcds_q87 | 12.2 | 27.3 | 2.241 | 2.245 |
| tpcds_q88 | 18.7 | 16.8 | 0.899 | 0.892 |
| tpcds_q89 | 7.6 | 10.8 | 1.416 | 1.435 |
| tpcds_q90 | 4.3 | 3.1 | 0.734 | 0.726 |
| tpcds_q91 | 6.0 | 5.7 | 0.948 | 0.937 |
| tpcds_q92 | 4.7 | 3.2 | 0.685 | 0.694 |
| tpcds_q93 | 10.5 | 9.5 | 0.913 | 0.916 |
| tpcds_q94 | 6.4 | 4.7 | 0.730 | 0.739 |
| tpcds_q95 | 16.4 | 12.1 | 0.741 | 0.740 |
| tpcds_q96 | 4.2 | 3.5 | 0.844 | 0.837 |
| tpcds_q97 | 10.3 | 10.3 | 1.007 | 0.983 |
| tpcds_q98 | 5.6 | 14.7 | 2.618 | 2.580 |
| tpcds_q99 | 8.3 | 968.1 | 116.514 | 118.185 |
| sum | 1487.0 | 3292.5 | 2.214 |
geomean ratio 1.214; geomean paired 1.220; 3 rounds x 5 iterations
A/A: geomean ratio 0.990; geomean paired 0.990; 3 rounds x 5 iterations
The largest TPC-DS gaps (q99, q62) use the same join modes but a different build side: VortexTable reports no column statistics, so DataFusion builds the hash table on catalog_sales rather than the dimension. On base, the VortexTable path on V1 took 3065 ms for q99, against 1000 ms here and 8 ms for the opener.
|
Any idea what's the cause of the worst regressions in this pr? |
|
These numbers compare two engine integrations, not before and after this PR. On base, the The worst gaps I looked at come from the integration, not the scan:
I haven't looked at TPC-H q10, q13 or q22 yet. The fixes for the two gaps above belong in |
PlannedDataSource becomes PlanDataSource and MultiDataSource becomes UnionDataSource (module vortex_scan::union), with DataSourceFactory now DeferredDataSource. ScanRequest gains target_partitions, which replaces PlanDataSource::with_max_partitions: engines that know their parallelism (DataFusion's target partitions) pass it, and a union divides it across the children it scans. Because the split count now depends on the request, a PlanDataSource is one selectable unit with index 0, and the Scan API docs say what an index names and how row_range and selection apply to each unit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Benchmarking VortexTable needed both --scan-v1 true and VORTEX_USE_SCAN_API=1, so it could not be compared with the listing table's V2 scan in one run. --scan-api true registers Vortex tables through the Scan API whatever --scan-v1 says; the environment variable keeps working. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Engines that consume the Scan API still scanned Vortex files with the V1 layout reader. PlanDataSource exposes any ScanInput as a DataSource whose single partition runs on the planner-driven ScanDriver, and VortexFile::data_source now uses it, falling back to the layout reader for files with list layouts, which the V2 executor cannot read yet. As the layout reader did, a planned file scan whose filter the footer statistics prove false reads nothing. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A PlanDataSource scan had one partition, so engines that parallelise by partition read a whole file on one worker. Unordered, unlimited scans now split the row range into near-equal partitions, sized by the new ScanRequest::target_partitions hint or the available parallelism, with at most one partition per started 65,536 rows. DataFusion passes its target partition count. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A split does not keep every partition above 65,536 rows: it makes at most one partition per started 65,536 rows, so 100,000 rows split into two halves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Records the split stack as this branch's base without changing its tree, so the pull request shows only the union layer. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scan I/O requests carry a SourceId and a segment target instead of a byte range, and one IoScheduler per session serves them. Placed sources read through the coalescing FileSegmentSource, serve the segments the footer read already fetched, share in-flight reads through SharedSegmentSource, and reuse a live registration across concurrent scans of one file; opaque sources read segments themselves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
a11813a to
2a14cbb
Compare
Benchmarking VortexTable needed both --scan-v1 true and VORTEX_USE_SCAN_API=1, so it could not be compared with the listing table's V2 scan in one run. --scan-api true registers Vortex tables through the Scan API whatever --scan-v1 says; the environment variable keeps working. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Engines that consume the Scan API still scanned Vortex files with the V1 layout reader. PlanDataSource exposes any ScanInput as a DataSource whose single partition runs on the planner-driven ScanDriver, and VortexFile::data_source now uses it, falling back to the layout reader for files with list layouts, which the V2 executor cannot read yet. As the layout reader did, a planned file scan whose filter the footer statistics prove false reads nothing. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A PlanDataSource scan had one partition, so engines that parallelise by partition read a whole file on one worker. Unordered, unlimited scans now split the row range into near-equal partitions, sized by the new ScanRequest::target_partitions hint or the available parallelism, with at most one partition per started 65,536 rows. DataFusion passes its target partition count. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Records the split stack as this branch's base without changing its tree, so the pull request shows only the union layer once it targets that branch. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Engines that keep a V1/V2 switch while they move onto the Scan API, and benchmarks comparing the two executors, need each one as a DataSource. layout_reader_data_source() returns the layout-reader source that data_source() already falls back to, with the same footer statistics, and data_source() now calls it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
#11626's union layer is superseded by PlanDataSource::from_children on #11665, which MultiFileDataSource::build_plan already builds. This merge keeps the branch's history and takes #11665 plus the table path change as its tree. Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Vortex benchmark target now names both axes: the integration (`file` through the engine's file format, `datasource` through DataFusion's `VortexTable`) and the scan engine (`v1` LayoutReader, `v2` plans). The scan engine was implicit per suite; suites that cannot run v2 yet are now only a constraint, not the selector. Results carry both axes as `integration` and `scan_engine` fields, and every combination but the file integration on the suite's default scan appends `[integration, scan]` to its format identity, so existing history continues. Validation rejects datasource-v2 (needs #11626) and DuckDB datasource (no table function yet). PR and develop run every valid combination on ClickBench and TPC-H SF1 NVMe; pr-full runs them on every suite, and nightly runs the other scans in companion jobs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
With `--scan-api true --scan-v1 false`, datafusion-bench registers a `VortexTable` over `MultiFileDataSource::build_plan`, so the datasource integration can run the V2 scan. The orchestrator now accepts datafusion datasource-v2 targets, and every-scan coverage runs them. Folds in #11626. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Plan(replacingLayoutReaders) #9062The DataFusion Scan API table path in
datafusion-bench(VortexTableover aMultiFileDataSource) only ran with the V1 scan, because it always calledMultiFileDataSource::build, which opens aLayoutReaderper file. #11665 gives the multi-file source a V2 form,build_plan, aPlanDataSourcewhose files open as the scan reaches them. WithVORTEX_USE_SCAN_API=1, the table path now picks between the two with--scan-v1, so the Scan API benchmark row can measureVortexTableon V2.This replaces the earlier version of this PR. That version added a
UnionDataSourceover per-file sources and row-range partitions.PlanDataSource::from_childrenin #11665 now covers the multi-file union natively, and row-range partitions were dropped from the stack.Behaviour change:
VORTEX_USE_SCAN_API=1together with--scan-v1 falseused to register a listing table and ignore the variable. It now registers the Scan API table on V2.Stacked on #11665. #11685 replaces the environment variable with a
--scan-apiflag; whichever lands second takes the other's change.🤖 Generated with Claude Code