Repository navigation
Share Arrow batch conversion across DataFusion scans - #11610
Conversation
Signed-off-by: Nicholas Gates <nick@nickgates.com>
3b1eeb7 to
cc273a3
Compare
Replaced by a newer auto-merge request
Merging this PR will degrade performance by 11.27%1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Auto-merge was disabled for another pull request in the stack
Use the shared driver for the DataFusion V2 file opener and remove its duplicate scheduler. The opener still waits for scan preparation so failures are reported as file-open errors, and Arrow conversion still runs on the task that produces each batch. When several chunks run on the same scheduler, such as with `scan_concurrency=0`, each chunk needs its own queue for pending reads. The shared driver used to give these chunks the same I/O session, so when one chunk finished and cleared the session, it could discard another chunk’s pending reads. Give each planner root its own session over the same file source, preserving the isolation DataFusion already had for inline chunks. Stacked on #11610. Part of #9062. This change was prepared with assistance from OpenAI Codex. --------- Signed-off-by: Nicholas Gates <nick@nickgates.com>
The file opener and the Scan API adapter both convert projected Vortex chunks into Arrow record batches. Sharing that conversion keeps the two paths consistent while each caller still chooses its target schema and runs the conversion on its existing CPU task.
Stacked on #11557.
Part of #2254.
This change was prepared with assistance from OpenAI Codex.