You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
But then I was thinking, if we are going to change the API anyways, why not go all the way with something fully statically typed and removing ConfigDefinition entirely:
/// DataFusion specific configuration names.pubenumConfigValue{/// Configuration option "datafusion.execution.target_partitions"TargetPartitions(usize),/// Configuration option for arbitrary user defined dataUserDefined{name:String,value:Option<ScalarValue>},
...}implConfigValue{/// Return the name of this configuration valuefn name(&self) -> &str{matchself{Self::TargetPartitions(_) => "datafusion.execution.target_partitions",
...}/// Return the human readable description for this configuration valuefn description(&self) -> &str{matchself{Self::TargetPartitions(_) =>
"Number of partitions for query execution. Increasing partitions can increase \ concurrency. Defaults to the number of cpu cores on the system.",
...}/// set the value of the configuration valuefn set_value(&mutself, new_value:ScalarValue) ->Result<()>{match(self, value){(Self::TargetPartitions(v),ScalarValue::UInt64(Some(new_value))) => *v = new_value,(Self::TargetPartitions(v), _) => returnErr("Expected uint64 for {} but got {:?}",self.name(), new_value)).
...
}
But before I go crank that through the process I wanted to get some feedback if that was a desirable way to go. I would retain the ability to store arbitrary name/value pairs in the metadata.
I'd like to be able to use Default::default() and struct update syntax if possible.
BTW, thank you @alamb for taking on this thankless task.
Recurring conflicts regarding multiple PRs each changing config in their own way is why I still have two of my own sitting in limbo. It will be nice to have this sorted - it should decrease the friction for everyone and PRs should flow faster.
One potential issue with struct update syntax, is any additional config is technically a breaking change. Unfortunately ..Default::default() doesn't work with #[non_exhaustive] as you might add a private field. See here. That being said the following is hardly the end of the world
let mut config = Config::default();
config.property = ...;
config
I don't have a strong opinion either way. The one thing I do like about @alamb original enum proposal is that is that we get a nice way to associate each config with a config key and description that would maybe a allow a more generic way of handling the decoding from CLI params and spitting out documentation/help text. Although you could probably do the same with a macro in the struct case.
Activity
alamb commentedon Dec 5, 2022
So I started bashing this out, with the incremental way like:
But then I was thinking, if we are going to change the API anyways, why not go all the way with something fully statically typed and removing
ConfigDefinitionentirely:But before I go crank that through the process I wanted to get some feedback if that was a desirable way to go. I would retain the ability to store arbitrary name/value pairs in the metadata.
What do you think @thinkharderdev @yahoNanJing @andygrove @avantgardnerio ?
tustvold commentedon Dec 14, 2022
If you're going to go for statically typed why not just go with just a struct? We could always add a HashMap for custom extensions as the final field.
avantgardnerio commentedon Dec 14, 2022
☝️ what he said :)
I'd like to be able to use
Default::default()and struct update syntax if possible.BTW, thank you @alamb for taking on this thankless task.
Recurring conflicts regarding multiple PRs each changing config in their own way is why I still have two of my own sitting in limbo. It will be nice to have this sorted - it should decrease the friction for everyone and PRs should flow faster.
tustvold commentedon Dec 14, 2022
One potential issue with struct update syntax, is any additional config is technically a breaking change. Unfortunately
..Default::default()doesn't work with#[non_exhaustive]as you might add a private field. See here. That being said the following is hardly the end of the worldthinkharderdev commentedon Dec 15, 2022
I don't have a strong opinion either way. The one thing I do like about @alamb original enum proposal is that is that we get a nice way to associate each config with a config key and description that would maybe a allow a more generic way of handling the decoding from CLI params and spitting out documentation/help text. Although you could probably do the same with a macro in the struct case.
andygrove commentedon Dec 15, 2022
I don't think I have a strong opinion either as long as we retain the current capabilities, specifically:
tustvold commentedon Dec 28, 2022
I had a play around and came up with this - https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=8c1ababec354db7118d3ff5ffd878f1e
I think it meets all of the above requirements, whilst being a "simple" structured type, and not requiring a proc-macro.
Let me know what you think, if people are happy with it I'd be happy to get something like it integrated with DataFusion.
alamb commentedon Dec 28, 2022
I think it looks good to me 👍
thinkharderdev commentedon Dec 28, 2022
Nice, seems like a good approach
Structify ConfigOptions (apache#4517)
Structify ConfigOptions (#4517) (#4771)