Repository navigation
Conversation
There was a problem hiding this comment.
I'm not very familiar with this code, and so you possibly want to wait for approval from someone else, but I think from a pure code standpoint this is fine.
However, I do notice that the current code is pretty consistently functional and this is effectively grafting an OOP abstraction on top of it. This isn't an inherently bad thing, but I wonder if you've given thought to impl Serializable for LogicalPlan or something similar, more in keeping with individual types knowing how to serialize themselves?
Potentially you could pass something like to Serializer instead of just &dyn FunctionRegistry
struct SerializerContext<'a> {
pub function_registry: Option<&dyn FunctionRegistry>,
pub extension_codec: Option<&dyn LogicalExtensionCodec>
}
FWIW I suspect this approach would make recursion into subqueries easier as well.
| // this trait was inherited from Ballista and was intended to support custom logical plans | ||
| // but that does not make sense in the context of this crate so would be good to remove | ||
| // this at some point |
There was a problem hiding this comment.
| // this trait was inherited from Ballista and was intended to support custom logical plans | |
| // but that does not make sense in the context of this crate so would be good to remove | |
| // this at some point | |
| /// This trait was inherited from Ballista and was intended to support custom logical plans | |
| /// but that does not make sense in the context of this crate so would be good to remove | |
| /// this at some point |
Perhaps a ticket also?
| //TODO note that we will need to also pass the function_registry along once we | ||
| // add support for subquery expressions in https://github.com/apache/arrow-datafusion/issues/2640 | ||
| let protobuf = match self.extension_codec { | ||
| Some(codec) => protobuf::LogicalPlanNode::try_from_logical_plan(plan, codec)?, |
There was a problem hiding this comment.
Is your plan to move this logic onto Serializer or to pass function_registry down?
|
Thanks for the review @tustvold. Based on feedback from @thinkharderdev in apache/datafusion-ballista#47 it seems that it would be preferable to expose this as a trait, based on I can take a look at this again at the weekend unless @liukun4515 wants to pick this up as part of #2640. |
|
marking as draft as there is future work planned here |
|
I am closing this since we have a working solution over in Ballista now (apache/datafusion-ballista#57) and I will let someone else pick up this task of improving the API here. I can try again later if nobody does pick this up. |
Which issue does this PR close?
N/A
Rationale for this change
New unified API for serializing plans and expressions.
What changes are included in this PR?
Are there any user-facing changes?
API change
Does this PR break compatibility with Ballista?
TBD