Repository navigation
[Java][C] sliced RecordBatch offset info is lost when imported from c-data #88
Description
Describe the bug, including details regarding any error messages, version, and platform.
Reproduced on latest arrow release (16.0)
When importing a sliced RecordBatch from c to java
On c side:
auto sliced_record_batch = original_record_batch->Slice(/*offset=*/8, /*length=*/2);
arrow::ExportRecordBatch(sliced_record_batch, arrow_array_ptr);
On java side:
ArrowArray arrowArray = ArrowArray.allocateNew(allocator);
Data.importIntoVectorSchemaRoot(allocator, arrowArray, vectorSchemaRoot, null);
The imported vectorSchemaRoot maintains the correct length(which is 2), but the offset info (which is 8) is not respected, hence the content of the imported vectorSchemaRoot points to the first 2 rows of the original_record_batch, while the desired content is sliced_record_batch.
I'm not familiar with arrow code, but it seems that the offset info is actually present in org.apache.arrow.c.ArrowArray.Snapshot, but org.apache.arrow.c.ArrayImporter ignores the offset in org.apache.arrow.c.ArrayImporter.doImport(ArrowArray.Snapshot)
Component(s)
Java
Activity
lidavidm commentedon May 16, 2024
Java doesn't really have a concept of slicing, so either this needs to copy the data (at least for things that can't easily be sliced, like the validity buffer) or it should error on import.
hellishfire commentedon May 16, 2024
I took a quick look at the slice() method of VectorSchemaRoot, which slices the data buffer, and slice or copy the validity buffer
lidavidm commentedon May 16, 2024
Yes, but that's different in intent than the C++ version (which just tracks an offset). Here it means we can't do a 0 copy import.
hellishfire commentedon May 16, 2024
I understand that, but IMO data correctness comes before performance, so a copy of validity buffer is tolerable in this case?
lidavidm commentedon May 16, 2024
CC @vibhatha
We also have to traverse recursively since child arrays can have their own offset
vibhatha commentedon May 16, 2024
I will try to reproduce this and evaluate the approach suggested by @lidavidm. I will probably need some time to try this out.
GH-704: Fix initialization of offset buffer when exporting VarChar ve…
apacheGH-704: Fix initialization of offset buffer when exporting VarC…
fix: zero sliced boolean offsets at every level before exporting to t…
fix: zero sliced boolean offsets at every level before exporting to t…