You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[C++][Dataset] Preserve order when writing dataset #26818
Currently, when writing a dataset, e.g. from a table consisting of a set of record batches, there is no guarantee that the row order is preserved when reading the dataset.
Small code example:
In [1]: importpyarrow.datasetasdsIn [2]: table = pa.table({"a": range(10)})
In [3]: table.to_pandas()
Out[3]:
a00112233445566778899In [4]: batches = table.to_batches(max_chunksize=2)
In [5]: ds.write_dataset(batches, "test_dataset_order", format="parquet")
In [6]: ds.dataset("test_dataset_order").to_table().to_pandas()
Out[6]:
a04152839465762738091
Although this might seem normal in SQL world, typical dataframe users (R, pandas/dask, etc) will expect a preserved row order.
Some applications might also rely on this, eg with dask you can have a sorted index column ("divisions" between the partitions) that would get lost this way (note, the dask parquet writer itself doesn't use pyarrow.dataset.write_dataset so isn't impacted by this.)
Some discussion about this started in #8305 (ARROW-9782), which changed to write all fragments to a single file instead of a file per fragment.
I am not fully sure what the best way to solve this, but IMO at least having the option to preserve the order would be good.
Weston Pace / @westonpace:
I deleted the link to ARROW-12873 because I don't know that "batch index" needs to rely on that arbitrary metadata mechanism (and, given that many nodes will need to manipulate it, I don't think it is arbitrary metadata)
Hi @westonpace,
Is there any updates on this?
Current "FileSystemDataset::Write" is implemented by a sequenced plan of scan, filter, project, write.
As referenced plan in #32991 , a batch_index has been added in the scanner and is used by "ordered_sink" to reorder exec_batches.
Should we consider implementing an "ordered" node that functions similar to ordered_sink without sinking? This node could be injected any place between scan and project.
I believe that the "ordered" node would be a more effective way to directly order the output of the "scan" node, providing a more flexible planning approach.
Just got burnt by the same issue as I was trying to re-encode a parquet file with a different rowgroup size. Even if there is no plan to fix that issue, it might be a good idea to add a warning in the documentation, which currently mentions nothing about order not being preserved: https://arrow.apache.org/docs/python/generated/pyarrow.dataset.Dataset.html#
Should we consider implementing an "ordered" node ...
Looks like we do not need to introduce a new node because the "write" can sequence exec batches already. For this to work, all we need is tell the "scan" to give batches and index (ImplicitOrdering) and the "write" will by default sequence the batches.
Activity
asfimport commentedon May 10, 2022
Weston Pace / @westonpace:
I deleted the link to ARROW-12873 because I don't know that "batch index" needs to rely on that arbitrary metadata mechanism (and, given that many nodes will need to manipulate it, I don't think it is arbitrary metadata)
hu6360567 commentedon Apr 10, 2024
Hi @westonpace,
Is there any updates on this?
Current "FileSystemDataset::Write" is implemented by a sequenced plan of
scan, filter, project, write.As referenced plan in #32991 , a batch_index has been added in the scanner and is used by "ordered_sink" to reorder exec_batches.
Should we consider implementing an "ordered" node that functions similar to ordered_sink without sinking? This node could be injected any place between
scanandproject.I believe that the "ordered" node would be a more effective way to directly order the output of the "scan" node, providing a more flexible planning approach.
jerryqhyu commentedon Jun 13, 2024
Could someone please solve this issue? This is clearly a bug in arrow, and it should least have an option to preserve order.
douglas-raillard-arm commentedon Jul 12, 2024
Just got burnt by the same issue as I was trying to re-encode a parquet file with a different rowgroup size. Even if there is no plan to fix that issue, it might be a good idea to add a warning in the documentation, which currently mentions nothing about order not being preserved:
https://arrow.apache.org/docs/python/generated/pyarrow.dataset.Dataset.html#
user529481 commentedon Jul 12, 2024
Related: #39030
Add comment to document that sorting in PyArrow is currently pointles…
EnricoMi commentedon Oct 18, 2024
Looks like we do not need to introduce a new node because the
"write"can sequence exec batches already. For this to work, all we need is tell the"scan"to give batches and index (ImplicitOrdering) and the"write"will by default sequence the batches.Here is the fix: #44470
GH-26818: [C++][Python] Preserve order when writing dataset multi-thr…
rok commentedon May 14, 2025
Issue resolved by pull request 44470
#44470
anjakefala commentedon May 15, 2025
@rok Yayyyy =D
apacheGH-26818: [C++][Python] Preserve order when writing dataset mul…
apacheGH-26818: [C++][Python] Preserve order when writing dataset mul…