Skip to content

GH-49340: [R] Preserve row order in write_dataset() - #49343

Merged
jonkeane merged 5 commits into
apache:mainfrom
marberts:preserve_order
Feb 24, 2026
Merged

jonkeane merged 5 commits into
apache:mainfrom
marberts:preserve_order

Conversation

@marberts

@marberts marberts commented Feb 20, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

write_dataset(df) need not preserve the row-ordering of df across partitions. The arrow C++ library was recently updated (since 21.0.0) so that row ordering can be preserved when writing across partitions. This is useful for cases where it is assumed that row-ordering is unchanged within each partition.

df <- tibble::tibble(x = 1:1.5e6, g = rep(1:15, each = 1e5))

df |>
  dplyr::group_by(g) |>
  arrow::write_dataset("test1", preserve_order = FALSE)

df |>
  dplyr::group_by(g) |>
  arrow::write_dataset("test2", preserve_order = TRUE)

test1 <- arrow::open_dataset("test1") |>
  dplyr::collect()

test2 <- arrow::open_dataset("test2") |>
  dplyr::collect()

# Current behavior.
all.equal(test1 |> sort_by(~ g), df)
#> [1] "Component \"x\": Mean relative difference: 0.0475804"

# Preserve order.
all.equal(test2 |> sort_by(~ g), df)
#> [1] TRUE

Created on 2026-02-20 with reprex v2.1.1

What changes are included in this PR?

Added an argument preserve_order to write_dataset() that sets FileSystemDatasetWriteOptions.preserve_order to true in the call to ExecPlan_Write().

Are these changes tested?

Partially. The change is small, so I haven't written unit tests. I can revisit this if necessary.

Are there any user-facing changes?

Yes, there is a new argument in write_dataset(). The default keeps the current behavior and the argument appears after all existing arguments, so the change in backwards compatible.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #49340 has been automatically assigned in GitHub to PR creator.

@jonkeane jonkeane left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution!

Would you mind please writing some tests for this behavior? Somewhere in https://github.com/apache/arrow/blob/main/r/tests/testthat/test-dataset-write.R (+ following similar patterns there) would be lovely.

@jonkeane jonkeane left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the tests, I have some suggestions about naming + slightly more idiomatic expectations.

It also looks like there are some cpp linting issues too: https://github.com/apache/arrow/actions/runs/22290480080/job/64535896409?pr=49343#step:6:42

Comment thread r/tests/testthat/test-dataset-write.R Resolved

@jonkeane jonkeane left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this! One question, but otherwise looks good.

Comment on lines +1035 to +1036
# expect_false(all(unordered_ds$x == df$x)) can fail on certain
# platforms, so is not tested.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, interesting — which platforms? I can also imagine that there are certain circumstances where it will be happen to wind up in the right order by chance too. It would be good to know where this failed though.

@jonkeane
jonkeane merged commit 376afb8 into apache:main Feb 24, 2026
16 checks passed
@marberts

marberts commented Feb 24, 2026 •

Copy link
Copy Markdown
Contributor Author

Awesome, thanks!

For your question, it seems like write_dataset() behaves differently on windows and ubuntu. In some cases row-order is kept on windows when preserve_order = FALSE (but not always). We can see this from the CI job when I had the test expect_false(all(unordered_ds$x == df$x)): it passed on ubuntu and windows with an older version of R, but failed on windows with a newer version of R (https://github.com/apache/arrow/actions/runs/22290480076/job/64535896168). I could reproduce this on my machine with R 4.5.2, with the same call to write_dataset() not preserving order on ubuntu and preserving order on windows.

@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 376afb8.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.

thisisnic pushed a commit to thisisnic/arrow that referenced this pull request Apr 6, 2026
…9343)

### Rationale for this change

`write_dataset(df)` need not preserve the row-ordering of `df` across partitions. The arrow C++ library was recently updated (since 21.0.0) so that row ordering can be preserved when writing across partitions. This is useful for cases where it is assumed that row-ordering is unchanged within each partition.

``` r
df <- tibble::tibble(x = 1:1.5e6, g = rep(1:15, each = 1e5))

df |>
  dplyr::group_by(g) |>
  arrow::write_dataset("test1", preserve_order = FALSE)

df |>
  dplyr::group_by(g) |>
  arrow::write_dataset("test2", preserve_order = TRUE)

test1 <- arrow::open_dataset("test1") |>
  dplyr::collect()

test2 <- arrow::open_dataset("test2") |>
  dplyr::collect()

# Current behavior.
all.equal(test1 |> sort_by(~ g), df)
#> [1] "Component \"x\": Mean relative difference: 0.0475804"

# Preserve order.
all.equal(test2 |> sort_by(~ g), df)
#> [1] TRUE
```

<sup>Created on 2026-02-20 with [reprex v2.1.1](https://reprex.tidyverse.org)</sup>

### What changes are included in this PR?

Added an argument `preserve_order` to `write_dataset()` that sets `FileSystemDatasetWriteOptions.preserve_order` to true in the call to `ExecPlan_Write()`.

### Are these changes tested?

Partially. The change is small, so I haven't written unit tests. I can revisit this if necessary.

### Are there any user-facing changes?

Yes, there is a new argument in `write_dataset()`. The default keeps the current behavior and the argument appears after all existing arguments, so the change in backwards compatible.

* GitHub Issue: apache#49340

Authored-by: Steve Martin <marberts@protonmail.com>
Signed-off-by: Jonathan Keane <jkeane@gmail.com>
Mottl pushed a commit to Mottl/arrow that referenced this pull request May 26, 2026
…9343)

### Rationale for this change

`write_dataset(df)` need not preserve the row-ordering of `df` across partitions. The arrow C++ library was recently updated (since 21.0.0) so that row ordering can be preserved when writing across partitions. This is useful for cases where it is assumed that row-ordering is unchanged within each partition.

``` r
df <- tibble::tibble(x = 1:1.5e6, g = rep(1:15, each = 1e5))

df |>
  dplyr::group_by(g) |>
  arrow::write_dataset("test1", preserve_order = FALSE)

df |>
  dplyr::group_by(g) |>
  arrow::write_dataset("test2", preserve_order = TRUE)

test1 <- arrow::open_dataset("test1") |>
  dplyr::collect()

test2 <- arrow::open_dataset("test2") |>
  dplyr::collect()

# Current behavior.
all.equal(test1 |> sort_by(~ g), df)
#> [1] "Component \"x\": Mean relative difference: 0.0475804"

# Preserve order.
all.equal(test2 |> sort_by(~ g), df)
#> [1] TRUE
```

<sup>Created on 2026-02-20 with [reprex v2.1.1](https://reprex.tidyverse.org)</sup>

### What changes are included in this PR?

Added an argument `preserve_order` to `write_dataset()` that sets `FileSystemDatasetWriteOptions.preserve_order` to true in the call to `ExecPlan_Write()`.

### Are these changes tested?

Partially. The change is small, so I haven't written unit tests. I can revisit this if necessary.

### Are there any user-facing changes?

Yes, there is a new argument in `write_dataset()`. The default keeps the current behavior and the argument appears after all existing arguments, so the change in backwards compatible.

* GitHub Issue: apache#49340

Authored-by: Steve Martin <marberts@protonmail.com>
Signed-off-by: Jonathan Keane <jkeane@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants