You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I would like to more cleanly separate the DDL (data definition) statements supported by DataFusion from the query execution.
Specifically, DataFusion offers basic support for "catalog" like operations such as registering external tables (CREATE EXTERNAL TABLE) , views (CREATE VIEW) and others. Some systems (like datafusion-cli or the tpch-benchmarks) use this basic support and others such as https://github.com/influxdata/influxdb_iox use DataFusion to query tables and views that are defined elsewhere.
The current support for catalog like operations is implemented as individual LogicalPlan variants such as LogicalPlan::CreateExternalTable
Adding support for a new statement (such as DROP TABLE, as in implement drop view #3267) results in a bunch of boiler plate that may not be needed
It is non-trivial to understand how to configure DataFusion so that it does not support CREATE VIEW or CREATE TABLE if that is not desired ( because, e.g. it would be a security hole - see this test):
Describe the solution you'd like
I would like to consolidate LogicalPlan::Create* and LogicalPlan::Drop* variants into a single LogicalPlan::DDL variant, and make a new DDLStatement enum like
Then the planner can still do things like optimize DML, replace placeholder tokens in DML/DQL, etc.
It also makes it so there is a consistent path in execution engines:
AST -> LogicalPlan -> PhysicalPlan
vs
AST -> [LogicalPlan, Statement] -> [query plan executor, statement executor]
however, if we merely group them as mentioned in this issue, then we can have the existing optimizer code focus on LogicalPlan::DQL and the rest of the code can keep functioning as it is.
Just my $0.02, interested to hear how this affects others...
Activity
[-]Consolidate DDL LogicalPlans[/-][+]Consolidate DDL LogicalPlans (refactor)[/+]tustvold commentedon Jan 3, 2023
Another option here would be to remove such DDL from
LogicalPlanentirely, and instead only surfaceDDLondatafusion_sql::Statement.The interface for
QueryPlannercould then be changed to usesql_parser::Queryinstead ofsql_parser::Statement.I'll have a play and see what I can come up with
[-]Consolidate DDL LogicalPlans (refactor)[/-][+]Consolidate DDL / Catalog manipulation LogicalPlans (refactor)[/+]avantgardnerio commentedon Apr 3, 2023
FWIW, doing so would undo the motivation behind the PR that kicked off this discussion. I think if we have:
LogicalPlan::DDL
LogicalPlan::DML(Insert(DQL), Update(DQL), Delete(DQL))
LogicalPlan::DQL(project, filter, etc)
Then the planner can still do things like optimize DML, replace placeholder tokens in DML/DQL, etc.
It also makes it so there is a consistent path in execution engines:
AST -> LogicalPlan -> PhysicalPlan
vs
AST -> [LogicalPlan, Statement] -> [query plan executor, statement executor]
however, if we merely group them as mentioned in this issue, then we can have the existing optimizer code focus on LogicalPlan::DQL and the rest of the code can keep functioning as it is.
Just my $0.02, interested to hear how this affects others...
alamb commentedon Jan 5, 2024
I believe all such plans are now part of LogicalPlan::Dml / LogicalPlan::Ddl so closing this issue
https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.LogicalPlan.html#variant.Dml
and https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.LogicalPlan.html#variant.Ddl