Skip to content

MINOR: Improve datafusion-proto API - #2670

Closed
andygrove wants to merge 6 commits into
apache:masterfrom
andygrove:improve-proto-api
Closed

andygrove wants to merge 6 commits into
apache:masterfrom
andygrove:improve-proto-api

Conversation

@andygrove

@andygrove andygrove commented May 31, 2022 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A

Rationale for this change

New unified API for serializing plans and expressions.

What changes are included in this PR?

let serializer = Serializer::new()
  .with_function_registry(...)
  .with_extension_codec(...);

let bytes = serializer.serialize_plan(...);

let bytes = serializer.serialize_expr(...);

Are there any user-facing changes?

API change

Does this PR break compatibility with Ballista?

TBD

@andygrove andygrove added the api change Changes the API exposed to users of the crate label May 31, 2022
@andygrove andygrove self-assigned this May 31, 2022
@andygrove
andygrove requested a review from alamb May 31, 2022 16:53

@tustvold tustvold left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not very familiar with this code, and so you possibly want to wait for approval from someone else, but I think from a pure code standpoint this is fine.

However, I do notice that the current code is pretty consistently functional and this is effectively grafting an OOP abstraction on top of it. This isn't an inherently bad thing, but I wonder if you've given thought to impl Serializable for LogicalPlan or something similar, more in keeping with individual types knowing how to serialize themselves?

Potentially you could pass something like to Serializer instead of just &dyn FunctionRegistry

struct SerializerContext<'a> {
  pub function_registry: Option<&dyn FunctionRegistry>,
  pub extension_codec: Option<&dyn LogicalExtensionCodec>
}

FWIW I suspect this approach would make recursion into subqueries easier as well.

Comment on lines +72 to +74
// this trait was inherited from Ballista and was intended to support custom logical plans
// but that does not make sense in the context of this crate so would be good to remove
// this at some point

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// this trait was inherited from Ballista and was intended to support custom logical plans
// but that does not make sense in the context of this crate so would be good to remove
// this at some point
/// This trait was inherited from Ballista and was intended to support custom logical plans
/// but that does not make sense in the context of this crate so would be good to remove
/// this at some point

Perhaps a ticket also?

//TODO note that we will need to also pass the function_registry along once we
// add support for subquery expressions in https://github.com/apache/arrow-datafusion/issues/2640
let protobuf = match self.extension_codec {
Some(codec) => protobuf::LogicalPlanNode::try_from_logical_plan(plan, codec)?,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is your plan to move this logic onto Serializer or to pass function_registry down?

@andygrove

Copy link
Copy Markdown
Member Author

Thanks for the review @tustvold. Based on feedback from @thinkharderdev in apache/datafusion-ballista#47 it seems that it would be preferable to expose this as a trait, based on AsLogicalPlan but we also need it to support function registries.

I can take a look at this again at the weekend unless @liukun4515 wants to pick this up as part of #2640.

@alamb

alamb commented Jun 1, 2022

Copy link
Copy Markdown
Contributor

marking as draft as there is future work planned here

@alamb
alamb marked this pull request as draft June 1, 2022 19:55
@andygrove

Copy link
Copy Markdown
Member Author

I am closing this since we have a working solution over in Ballista now (apache/datafusion-ballista#57) and I will let someone else pick up this task of improving the API here. I can try again later if nobody does pick this up.

@andygrove andygrove closed this Jun 4, 2022
@andygrove
andygrove deleted the improve-proto-api branch January 27, 2023 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api change Changes the API exposed to users of the crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants