You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Different schemas when inferring from local system in different OS #5779
When inferring a schema, the list_all_files uses an object store to list the files. No sorting is passed.
When the object store is a LocalFileSystem, there isn't an insurance of any file sorting (the return list of a macOs has a different sort of windows). This means that the inferred schema can be different for the same set of files.
We contact the object store (apache/arrow-rs-object-store#178) that point it out that the solution should be implemented in the caller of the method, applying a sort of any type, to maintain consistency between file systems.
To Reproduce
Having two parquet files in the filesystem with the schema:
#[tokio::test]
async fn infer_schema() {
let path = ListingTableUrl::parse("./files").unwrap();
let ctx = SessionContext::new();
let state = ctx.state();
let options = ListingOptions::new(Arc::new(ParquetFormat::default()));
let schema = options.infer_schema(&state, &path).await.unwrap();
schema.fields.iter().for_each(|field| println!("{0}", field.name()));
}
the result in macOs Ventura:
description
code
year
the first file pickup was the file3.parquet
and using windows
year
code
description
the first file pickup was the file1.parquet
Expected behavior
The same schema independently the OS where the code is run. A sort should be forced or at least given the possibility of passing a sort function
changed the title [-]Different schemas when inferring schema from folder[/-][+]Different schemas when inferring schema from local system in different OS[/+]on Mar 29, 2023
changed the title [-]Different schemas when inferring schema from local system in different OS[/-][+]Different schemas when inferring from local system in different OS[/+]on Mar 29, 2023
Activity
[-]Different schemas when inferring schema from folder[/-][+]Different schemas when inferring schema from local system in different OS[/+][-]Different schemas when inferring schema from local system in different OS[/-][+]Different schemas when inferring from local system in different OS[/+]thomas-k-cameron commentedon Jun 10, 2023
I just created a PR for this issue. There are still some work to do (e.g. tests) but I hope it works!