Skip to content

Share Arrow batch conversion across DataFusion scans - #11610

Merged
gatesn merged 1 commit into
developfrom
ngates/shared-scan-api
Nov 18, 2026
Merged

gatesn merged 1 commit into
developfrom
ngates/shared-scan-api

Conversation

@gatesn

@gatesn gatesn commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

The file opener and the Scan API adapter both convert projected Vortex chunks into Arrow record batches. Sharing that conversion keeps the two paths consistent while each caller still chooses its target schema and runs the conversion on its existing CPU task.

Stacked on #11557.
Part of #2254.

This change was prepared with assistance from OpenAI Codex.

@gatesn gatesn added the changelog/chore A trivial change label Nov 18, 2026
@gatesn
gatesn added this pull request to stack #11612 November 18, 2026 19:50
Base automatically changed from ngates/consolidate-scan-engine to develop November 18, 2026 20:08
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn force-pushed the ngates/shared-scan-api branch from 3b1eeb7 to cc273a3 Compare November 18, 2026 20:15
@gatesn
gatesn marked this pull request as ready for review November 18, 2026 20:17
@gatesn
gatesn requested a review from connortsui20 November 18, 2026 20:17
@gatesn
gatesn enabled stacked auto-merge (squash) November 18, 2026 20:17
stacked auto-merge was automatically disabled November 18, 2026 20:18

Replaced by a newer auto-merge request

@gatesn
gatesn enabled stacked auto-merge (squash) November 18, 2026 20:18
@codspeed

codspeed Bot commented Nov 18, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 11.27%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

❌ 1 regressed benchmark
✅ 1974 untouched benchmarks
⏩ 53 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime threshold_avx512[80] 2.1 µs 2.4 µs -11.27%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ngates/shared-scan-api (cc273a3) with develop (d537380)

Open in CodSpeed

Footnotes

  1. 53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

stacked auto-merge was automatically disabled November 18, 2026 20:38

Auto-merge was disabled for another pull request in the stack

@gatesn
gatesn merged commit 452f24a into develop Nov 18, 2026
128 of 129 checks passed
@gatesn
gatesn deleted the ngates/shared-scan-api branch November 18, 2026 20:38
gatesn added a commit that referenced this pull request Nov 19, 2026
Use the shared driver for the DataFusion V2 file opener and remove its
duplicate scheduler. The opener still waits for scan preparation so
failures are reported as file-open errors, and Arrow conversion still
runs on the task that produces each batch.

When several chunks run on the same scheduler, such as with
`scan_concurrency=0`, each chunk needs its own queue for pending reads.
The shared driver used to give these chunks the same I/O session, so
when one chunk finished and cleared the session, it could discard
another chunk’s pending reads. Give each planner root its own session
over the same file source, preserving the isolation DataFusion already
had for inline chunks.

Stacked on #11610.
Part of #9062.

This change was prepared with assistance from OpenAI Codex.

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants