You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Rust] [Parquet] ArrowReader fails on some timestamp types #24454
> SELECTtpep_pickup_datetimeFROMtaxiLIMIT1;
General("InvalidArgumentError(\"column types must match schema types, expected Timestamp(Microsecond, None) but found UInt64 at column index 0\")")
The parquet reader detects this schema when reading from the file:
When the Parquet arrow reader creates the record batch, the following validation logic fails:
foriin0..columns.len() {
ifcolumns[i].len() != len {
returnErr(ArrowError::InvalidArgumentError(
"all columns in a record batch must have the same length".to_string(),
));
}
ifcolumns[i].data_type() != schema.field(i).data_type() {
returnErr(ArrowError::InvalidArgumentError(format!(
"column types must match schema types, expected {:?} but found {:?} at column index {}",
schema.field(i).data_type(),
columns[i].data_type(),
i)));
}
}
Activity
asfimport commentedon Mar 29, 2020
Andy Grove / @andygrove:
@liurenjie1024
[~sunchao]I may need some help with this one.asfimport commentedon Mar 30, 2020
Renjie Liu / @liurenjie1024:
I think the root cause is here https://github.com/apache/arrow/blob/master/rust/parquet/src/arrow/array_reader.rs#L220
The array reader only did conversion of data buffer, but left data type incorrect. I'll submit a PR to fix it this week.
asfimport commentedon Oct 23, 2020
Neville Dipale / @nevi-me:
We'll get this fixed as part of the Parquet <> Arrow IO work
asfimport commentedon Dec 15, 2020
Neville Dipale / @nevi-me:
I'm closing this, as the reader in the current master supports converting types correctly.