You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Handle Serde for Custom ScalarUDFImpl traits #8706
Nice! It would be very useful to be able to handle serde as well for custom implementations (perhaps in a different PR?). I think this could fit relatively easily into LogicalExtensionCodec
eg, we just use the name and assume wherever it is being deserialized will just have a registry where it can look up the scalar function definition by name.
But ideally we would be able to serialize a custom scalar function that has some sort of associated state. For example, a regex scalar function that actually contains the compiled regex in it's struct definition like:
struct MyRegexUdf {
compiled_regex: Vec<u8> // just assume we have some serialization of the regex state machine here
}
impl ScalarUDFImpl for MyRegexUdf {
fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
// do something with compiled regex here
}
}
Currently the mechanism that is used for this kind of thing is to define a custom LogicalExtensionCodec. So here I think we would add methods to that trait like
impl LogicalExtensionCodec for DefaultLogicalExtensionCodec {
fn try_encode_scalar_udf(
&self,
_node: Arc<dyn ScalarUdfImpl>,
_buf: &mut Vec<u8>,
) -> Result<()> {
not_impl_err!("LogicalExtensionCodec is not provided")
};
fn try_decode_scalar_udf(
&self,
_buf: &[u8],
_ctx: &SessionContext,
) -> Result<Arc<dyn ScalarUDFImpl>> {
not_impl_err!("LogicalExtensionCodec is not provided")
}
}
So then I would be able to define my own UDFs that contain internal state and then define an extension codec like
struct MyLogicalExtensionCodec;
impl LogicalExentionCodec for MyLogicalExtensionCodec {
fn try_encode_scalar_udf(
&self,
node: Arc<dyn ScalarUdfImpl>,
buf: &mut Vec<u8>,
) -> Result<()> {
if let Some(regex_udf) = node.as_any().downcast_ref::<MyRegexUdf> {
let proto = MyRegexUdfProto {
compiled_regex: regex.compiled_regex.clone()
}
proto.encode(buf)?;
Ok(())
} else {
not_impl_err!("LogicalExtensionCodec is not provided")
}
};
fn try_decode_scalar_udf(
&self,
buf: &[u8],
_ctx: &SessionContext,
) -> Result<Arc<dyn ScalarUDFImpl>> {
if let Ok(proto) = MyRegexUdfProto::decode(buf) {
Ok(Arc::new(MyRegexUdf { compiled_regex: proto.compiled_regex)))
} else {
not_impl_err!("LogicalExtensionCodec is not provided")
}
}
}
However, this doesn't play very nicely with how the serde is currently defined because we have no way to get a LogicalExtensionCodec in our impl TryFrom<&Expr> for protobuf::LogicalExprNode which we would need
Perhaps this could be incorporated into FunctionRegistry and we have a serialize_expr(expr: &Expr, registry: &dyn FunctionRegistry) similar to how we do the deserialization?
@thinkharderdev Much appreciated! I understand the issue now.
Another question is there some example code for deserialization so that I can refer to them to think how to write serialize_expr function?
@thinkharderdev Much appreciated! I understand the issue now. Another question is there some example code for deserialization so that I can refer to them to think how to write serialize_expr function?
Edit: On second thought we can probably just extend LogicalExtensionCodec and PhysicalExtensionCodec
Hey @yyy1000, looking at this a bit and I think what we want here is:
Have ScalarUDFExprNode take an optional opaque payload to potentially contain the serialized function:
This would be mostly unchanged from the existing TryFrom implementation except for handling of Expr::ScalarFunction/AggregateFunction/WindowFunction. We would handle Expr::ScalarFunction something like:
A question I have is, replace Replace impl TryFrom<&Expr> for protobuf::LogicalExprNode with a free function may lead a lot of places like expr.try_into() to serialize_expr(expr, codec), would this be OK for the code base? @thinkharderdev
A question I have is, replace Replace impl TryFrom<&Expr> for protobuf::LogicalExprNode with a free function may lead a lot of places like expr.try_into() to serialize_expr(expr, codec), would this be OK for the code base? @thinkharderdev
Yeah I think so. @alamb do you see any issues with that?
A question I have is, replace Replace impl TryFrom<&Expr> for protobuf::LogicalExprNode with a free function may lead a lot of places like expr.try_into() to serialize_expr(expr, codec), would this be OK for the code base? @thinkharderdev
Yeah I think so. @alamb do you see any issues with that?
I don't see any specific issue and I don't think it would affect users of the crate much -- I don't think they typically use protobuf encoding directly, but rather go through the higher level apis like Expr::to_bytes():
Activity
yyy1000 commentedon Jan 24, 2024
I'd also like to work on this. 😃
yyy1000 commentedon Jan 25, 2024
I need some guide on this. 🤔
It seems ScalarUDF can handle Serde now, what would this PR want to implement?
https://github.com/apache/arrow-datafusion/blob/7a0af5be2323443faa75cc5876651a72c3253af8/datafusion/proto/src/logical_plan/from_proto.rs#L1818-L1826
alamb commentedon Jan 25, 2024
Maybe @thinkharderdev can comment -- perhaps nothing is needed?
yyy1000 commentedon Jan 25, 2024
Aha, maybe I can help other issues first. 😄
thinkharderdev commentedon Jan 25, 2024
Hey @yyy1000 I think there is some work to do here. Currently the serialization for udf looks like
eg, we just use the
nameand assume wherever it is being deserialized will just have a registry where it can look up the scalar function definition by name.But ideally we would be able to serialize a custom scalar function that has some sort of associated state. For example, a regex scalar function that actually contains the compiled regex in it's struct definition like:
Currently the mechanism that is used for this kind of thing is to define a custom
LogicalExtensionCodec. So here I think we would add methods to that trait likeSo then I would be able to define my own UDFs that contain internal state and then define an extension codec like
However, this doesn't play very nicely with how the serde is currently defined because we have no way to get a
LogicalExtensionCodecin ourimpl TryFrom<&Expr> for protobuf::LogicalExprNodewhich we would needthinkharderdev commentedon Jan 25, 2024
Perhaps this could be incorporated into
FunctionRegistryand we have aserialize_expr(expr: &Expr, registry: &dyn FunctionRegistry)similar to how we do the deserialization?yyy1000 commentedon Jan 29, 2024
@thinkharderdev Much appreciated! I understand the issue now.
Another question is there some example code for
deserializationso that I can refer to them to think how to writeserialize_exprfunction?thinkharderdev commentedon Feb 26, 2024
Edit: On second thought we can probably just extend
LogicalExtensionCodecandPhysicalExtensionCodecHey @yyy1000, looking at this a bit and I think what we want here is:
ScalarUDFExprNodetake an optional opaque payload to potentially contain the serialized function:Would do a similar thing for
AggregateUDFExprNodeandWindowExprNodeLogicalExtensionCodecandPhysicalExtensionCodecto support custom serde on UDF/UDAF/UDWFimpl TryFrom<&Expr> for protobuf::LogicalExprNodewith a free functionThis would be mostly unchanged from the existing
TryFromimplementation except for handling ofExpr::ScalarFunction/AggregateFunction/WindowFunction. We would handleExpr::ScalarFunctionsomething like:parse_exprwe try and use the extension codec to deserialize the function iffun_definitionis present:If you don't have bandwidth now to work on this let me know, My team can take this up as we are hoping to be able to use this functionality soon.
yyy1000 commentedon Feb 27, 2024
Thanks so much for your detailed instructions! @thinkharderdev
I can try to implement it now and will seek your help if needed :)
yyy1000 commentedon Feb 27, 2024
A question I have is, replace Replace
impl TryFrom<&Expr> for protobuf::LogicalExprNodewith a free function may lead a lot of places likeexpr.try_into()toserialize_expr(expr, codec), would this be OK for the code base? @thinkharderdevthinkharderdev commentedon Feb 27, 2024
Yeah I think so. @alamb do you see any issues with that?
yyy1000 commentedon Feb 27, 2024
Also I wonder how to deal with some other places which don't need a
LogicalExtensionCodecliketo_byteshttps://github.com/apache/arrow-datafusion/blob/c439bc73b6a9ba9efa4c8a9b5d2fb6111e660e74/datafusion/proto/src/bytes/mod.rs#L87-L92. I think add a param of type
LogicalExtensionCodecto the function will work, and what to do when calling this method, should I initialize a implLogicalExtensionCodec? 🤔alamb commentedon Feb 28, 2024
I don't see any specific issue and I don't think it would affect users of the crate much -- I don't think they typically use protobuf encoding directly, but rather go through the higher level apis like
Expr::to_bytes():https://github.com/apache/arrow-datafusion/blob/e62240969135e2236d100c8c0c01546a87950a80/datafusion/proto/src/lib.rs#L52-L68