Skip to content

[C++][Dataset] Preserve order when writing dataset #26818

Closed

Description

@asfimport

Currently, when writing a dataset, e.g. from a table consisting of a set of record batches, there is no guarantee that the row order is preserved when reading the dataset.

Small code example:

In [1]: import pyarrow.dataset as ds

In [2]: table = pa.table({"a": range(10)})

In [3]: table.to_pandas()
Out[3]: 
   a
0  0
1  1
2  2
3  3
4  4
5  5
6  6
7  7
8  8
9  9

In [4]: batches = table.to_batches(max_chunksize=2)

In [5]: ds.write_dataset(batches, "test_dataset_order", format="parquet")

In [6]: ds.dataset("test_dataset_order").to_table().to_pandas()
Out[6]: 
   a
0  4
1  5
2  8
3  9
4  6
5  7
6  2
7  3
8  0
9  1

Although this might seem normal in SQL world, typical dataframe users (R, pandas/dask, etc) will expect a preserved row order.
Some applications might also rely on this, eg with dask you can have a sorted index column ("divisions" between the partitions) that would get lost this way (note, the dask parquet writer itself doesn't use pyarrow.dataset.write_dataset so isn't impacted by this.)

Some discussion about this started in #8305 (ARROW-9782), which changed to write all fragments to a single file instead of a file per fragment.

I am not fully sure what the best way to solve this, but IMO at least having the option to preserve the order would be good.

cc @bkietz

Reporter: Joris Van den Bossche / @jorisvandenbossche
Watchers: Rok Mihevc / @rok

Related issues:

Note: This issue was originally created as ARROW-10883. Please see the migration documentation for further details.

Activity

  1. asfimport commented on May 10, 2022

    @asfimport
    Author

    Weston Pace / @westonpace:
    I deleted the link to ARROW-12873 because I don't know that "batch index" needs to rely on that arbitrary metadata mechanism (and, given that many nodes will need to manipulate it, I don't think it is arbitrary metadata)

  2. added this to the 11.0.0 milestone on Jan 11, 2023
  3. removed this from the 11.0.0 milestone on Jan 11, 2023
  4. hu6360567 commented on Apr 10, 2024

    @hu6360567
    Contributor

    Hi @westonpace,
    Is there any updates on this?
    Current "FileSystemDataset::Write" is implemented by a sequenced plan of scan, filter, project, write.
    As referenced plan in #32991 , a batch_index has been added in the scanner and is used by "ordered_sink" to reorder exec_batches.
    Should we consider implementing an "ordered" node that functions similar to ordered_sink without sinking? This node could be injected any place between scan and project.
    I believe that the "ordered" node would be a more effective way to directly order the output of the "scan" node, providing a more flexible planning approach.

  5. jerryqhyu commented on Jun 13, 2024

    @jerryqhyu

    Could someone please solve this issue? This is clearly a bug in arrow, and it should least have an option to preserve order.

  6. douglas-raillard-arm commented on Jul 12, 2024

    @douglas-raillard-arm

    Just got burnt by the same issue as I was trying to re-encode a parquet file with a different rowgroup size. Even if there is no plan to fix that issue, it might be a good idea to add a warning in the documentation, which currently mentions nothing about order not being preserved:
    https://arrow.apache.org/docs/python/generated/pyarrow.dataset.Dataset.html#

  7. user529481 commented on Jul 12, 2024

    @user529481

    Related: #39030

  8. EnricoMi commented on Oct 18, 2024

    @EnricoMi
    Collaborator

    Should we consider implementing an "ordered" node ...

    Looks like we do not need to introduce a new node because the "write" can sequence exec batches already. For this to work, all we need is tell the "scan" to give batches and index (ImplicitOrdering) and the "write" will by default sequence the batches.

    Here is the fix: #44470

  9. added a commit that references this issue on May 14, 2025
  10. rok commented on May 14, 2025

    @rok
    Member

    Issue resolved by pull request 44470
    #44470

  11. added this to the 21.0.0 milestone on May 14, 2025
  12. anjakefala commented on May 15, 2025

    @anjakefala
    Contributor

    @rok Yayyyy =D

  13. added 2 commits that reference this issue on May 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions