Skip to content

Transforming LogicalPlan::Explain using Treenode::transform fails unexpectedly #8396

Closed

Description

@trueleo

Describe the bug

How did i discover this issue

In parseable we have external box for selecting timerange beside query. I want this external filter to be directly applied to any Table referenced in SQL so that this filter expression is available at scan time.

What is the bug ?

In the project I converted the SQL into LogicalPlan and used TreeNode::transform method to mutate the plan to include the external time range filter.

If the plan is Explain, transformation fails with error message. Otherwise it seems to work just fine

 Plan("Invalid EXPLAIN command. Expression is empty")

To Reproduce

minimal program to reproduce this

use std::sync::Arc;
use datafusion::{
    arrow::{
        array::BooleanArray,
        datatypes::{DataType, Field, Fields, Schema},
        record_batch::RecordBatch,
    },
    common::tree_node::{Transformed, TreeNode},
    datasource::{provider_as_source, MemTable},
    logical_expr::{col, Expr, Filter, LogicalPlan, LogicalPlanBuilder},
    scalar::ScalarValue,
};

fn main() {
    // bit of boilerplate code 
    let schema = Arc::new(Schema::new(Fields::from(vec![Field::new(
        "a",
        DataType::Boolean,
        false,
    )])));
    let rb = RecordBatch::try_new(
        schema.clone(),
        vec![Arc::new(BooleanArray::from(vec![true]))],
    )
    .unwrap();
    let table = Arc::new(MemTable::try_new(schema, vec![vec![rb]]).unwrap());

    // example logical plan
    let plan = LogicalPlanBuilder::scan("table", provider_as_source(table), None)
        .unwrap()
        // new explain is created here.
        .explain(false, false)
        .unwrap()
        .build()
        .unwrap();

    // example external filter that i want to add
    let external_filter = col("a").eq(Expr::Literal(ScalarValue::Boolean(Some(true))));

    // after transformation, because plan is not the same anymore,
    // the parent plan is built again with call to LogicalPlan::with_new_inputs -> with_new_exprs
    plan.transform(&|plan| match plan {
        LogicalPlan::TableScan(table) => {
            let filter = Filter::try_new(
                external_filter.clone(),
                Arc::new(LogicalPlan::TableScan(table)),
            )
            .unwrap();
            Ok(Transformed::Yes(LogicalPlan::Filter(filter)))
        }
        x => Ok(Transformed::No(x)),
    })
    // This panics because LogicalPlan::with_new_exprs fail. Explain yeilds empty expression
    // Panic message - Invalid EXPLAIN command. Expression is empty
    .unwrap();

    // run the plan in datafusion session
}

Expected behavior

Creates a new explain plan without any issue.

Additional context

Error orignates in LogicalPlan::new_with_exprs

Activity

  1. alamb commented on Dec 1, 2023

    @alamb
    Contributor

    Thank you for the report -- I agree explain plan may need special handling

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions