You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Predicate still has cast when comparing Timestamp(Nano, None) to a timestamp literal, so can't be pushed down or used for pruning #3938
Describe the bug
Comparing a Timestamp(Nanosecond, None) column to a timestamp literal is important for IOx and can be used to potentially prune significant amounts of data and pushed down to scans.
Specifically, to get the last hours of data you can use a predicate like:
To Reproduce
Add this test to the optimizer-integration test:
#[test]fntimestamp_nano_ts_none_predicates() -> Result<()>{let sql = "SELECT col_int32 FROM test WHERE col_ts_nano_none < (now() - interval '1 hour')";let plan = test_sql(sql)?;// a scan should have the now()... predicate folded to a single// constant and compared to the column without a cast so it can be// pushed down / prunedlet expected = "Projection: test.col_int32\n Filter: test.col_ts_nano_utc < TimestampNanosecond(1666612093000000000, Some(\"UTC\"))\\n TableScan: test projection=[col_int32, col_ts_nano_none]";assert_eq!(expected, format!("{:?}", plan));Ok(())}
It is not clear to me if the better fix would be to provide Timestamp(Nanos, UTC) rather than Timestamp(Nanos, None) in this case. However, DataFusion could definitely do a better job with unwrapping the cast
Activity
alamb commentedon Nov 8, 2022
It turns out I accidentally marked this as completed when it is still broken