You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Projection pushdown removes unqualified column names even when they are used #617
After that PR the projection pushdown logic decides that since the col1, col2 and col3 references don't have the table qualifier t1 on them, they are removed and the optimized plan looks like
🤔 I think this may be more complicated than I thought - I am not sure if I am creating the plans incorrectly or if projection pushdown is doing the wrong thing. I need to study this some more
hmm... the projection push down should be able to handle unqualified columns without problem, i.e. the current design doesn't require clients to add qualifiers to all the fields. Do you have a reproducible example i can play with? I definitely didn't test the change with extension nodes.
Activity
alamb commentedon Jun 24, 2021
FYI @houqp -- I believe I have found the problem, but now I need to sort out one other failure before I can make a PR
alamb commentedon Jun 24, 2021
🤔 I think this may be more complicated than I thought - I am not sure if I am creating the plans incorrectly or if projection pushdown is doing the wrong thing. I need to study this some more
alamb commentedon Jun 24, 2021
So at least initially the answer appears to be "I should be creating fully qualified
Exprs (with relation name)"Here is the change I needed to make in IOx:
This was a non obvious failure more however, and I can probably figure out a better way to detect such errors. Anyhow, closing for now
houqp commentedon Jun 24, 2021
hmm... the projection push down should be able to handle unqualified columns without problem, i.e. the current design doesn't require clients to add qualifiers to all the fields. Do you have a reproducible example i can play with? I definitely didn't test the change with extension nodes.
alamb commentedon Jun 24, 2021
@houqp I will get one
alamb commentedon Jun 25, 2021
Here is a reproducer (in projection_push_down.rs) -- it can also be found on https://github.com/alamb/arrow-datafusion/tree/alamb/repro_projection_pruning
It fails in the following way:
The problem is that the
TableScanis only scanning one column, even though theProjectionis using two columns.I will put up a PR with code to fix the problem
alamb commentedon Jun 25, 2021
#619
Expose `register_listing_table` (apache#618)