Repository navigation
Expose physical plan optimizer #1238
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I want to add custom physical plan optimizer rules, written in rust, to datafusion-python. Now that we have the ability to share execution plans across the FFI boundary we should be able to expose this interface.
Describe the solution you'd like
- Implement FFI upstream
datafusion - Expose
with_physical_optimizer_rulesand/orwith_physical_optimizer_rule
Describe alternatives you've considered
None
Additional context
This will support things like https://github.com/datafusion-contrib/datafusion-distributed in datafusion-python
Activity
timsaucer commentedon Sep 17, 2025
Upon deeper evaluation, this seems non-trivial since the optimizer won't understand what kinds of plans its getting. I suspect we need to go a route of serializing/deserializing the plans.
milenkovicm commentedon Oct 21, 2025
I guess this is the same issue like ballista support
renato2099 commentedon Jan 12, 2026
couldn't substrate.io be considered here as well ?