Skip to content

[C++] write_dataset: max_open_files does not close least recently used file #45038

Open

Description

@xWaita

Describe the bug, including details regarding any error messages, version, and platform.

On the latest version of pyarrow (18.0.0), according to the docs of pyarrow.dataset.write_dataset, when max_open_files is reached, the least recently used file will be closed. (https://arrow.apache.org/docs/python/generated/pyarrow.dataset.write_dataset.html)

However in the C++ code of dataset writer, what actually happens is the largest file is closed. (https://github.com/apache/arrow/blob/main/cpp/src/arrow/dataset/dataset_writer.cc#L656)

For long running dataset writes with many partitions, this means that once max open files is saturated, over time the set of open files will trend towards being smaller and smaller in size. Recently opened files can begin to be closed prematurely, while old open files can hang around and take up a file slot.

It would probably make sense for C++ behaviour to be brought in line with the docs, which should be to close the least recently used file rather than the largest file, with the rationale that the least recently used file is more likely to be finished permanently.

A solution could be to store the last write time whenever DatasetWriterDirectoryQueue::StartWrite() is called (maybe someone more knowledgeable about the codebase can suggest somewhere better), and use that to replace DatasetWriter::TryCloseLargestFile() with something like DatasetWriter::TryCloseLeastRecentlyUsedFile().

Component(s)

C++, Python

Activity

  1. changed the title [-]write_dataset max_open_files does not close oldest file[/-] [+]write_dataset: max_open_files does not close least recently used file[/+] on Dec 17, 2024
  2. raulcd commented on Dec 17, 2024

    @raulcd
    Member
  3. pitrou commented on Dec 17, 2024

    @pitrou
    Member

    It would probably make sense for C++ behaviour to be brought in line with the docs, which should be to close the least recently used file rather than the largest file, with the rationale that the least recently used file is more likely to be finished permanently.

    Thanks for the report, and yes, I agree this would be a good idea.

  4. pitrou commented on Dec 17, 2024

    @pitrou
    Member

    Hmm, the current behavior seems to be deliberate according to the PR description:

    FileSystemDatasetWriteOptions now has a max_open_files setting (ARROW-12321) which prevents opening too many files. Instead the writer will apply backpressure on the scanner while also closing the open file with the greatest # of rows already written (then resume writing once the file is closed).

    @westonpace Do you happen to remember why the option description doesn't match the intended behavior? Is it just a mistake in the docstring?

  5. mapleFU commented on Dec 17, 2024

    @mapleFU
    Member

    This idea LGTM, would you like to implement this? Or I'll check it then

  6. changed the title [-]write_dataset: max_open_files does not close least recently used file[/-] [+][C++] write_dataset: max_open_files does not close least recently used file[/+] on Dec 17, 2024
  7. xWaita commented on Dec 17, 2024

    @xWaita
    Author

    Thanks for the replies! I can work on it

  8. mapleFU commented on Dec 17, 2024

    @mapleFU
    Member

    why the option description doesn't match the intended behavior? Is it just a mistake in the docstring?

    I guess maybe using same strategy when memory overlimit and fd over limit? Since when we want to reclaim memory, close the largest file ( or k largest file) is usally a good way?

  9. pitrou commented on Dec 17, 2024

    @pitrou
    Member

    Before working on anything, we should decide which behavior is most desirable. If you want your dataset files to have relatively similar sizes, then it makes sense to close the largest files first.

  10. xWaita commented on Dec 17, 2024

    @xWaita
    Author

    There is a secondary write_dataset parameter in max_rows_per_file that can be used to control how large each file will get.

    With the current implementation of max_open_files and writing a dataset with many partitions, rather than seeing file sizes remaining similar, you'd actually see file sizes steadily trend downwards due to small partition files that remain open, while newer partition files compete to stay under the file size of those smaller partitions.

  11. pitrou commented on Dec 17, 2024

    @pitrou
    Member

    With the current implementation of max_open_files and writing a dataset with many partitions, rather than seeing file sizes remaining similar, you'd actually see file sizes steadily trend downwards due to small partition files that remain open, while newer partition files compete to stay under the file size of those smaller partitions.

    Ah, that may be the case if you have many partitions (more than max_open_files) that are very rarely hit?

  12. xWaita commented on Dec 18, 2024

    @xWaita
    Author

    yeah exactly, and I believe this would be a very common usecase, i.e. if you were to process/write a large file where the partition column is sorted (take a date partitioned file for example), new partitions would constantly be created while old partitions would never be touched again

  13. added a commit that references this issue on Dec 31, 2024
  14. ryancasburn-KAI commented on Mar 22, 2025

    @ryancasburn-KAI

    As @mapleFU pointed out, it seems like this one parameter (max_open_files) is being used for two purposes, keep total file descriptors under a limit and keep memory usage reasonable.

    Perhaps a second parameter should be added: max_open_rows. This would control memory usage while max_open_files controls fd use.

    If max_open_rows is reached, the largest file is written (which would drop the number of open rows and memory usage the most). If max_open_files is reached, the least recently used file is written. This gives people the option to tune the performance as they need.

    max_open_rows could be defaulted to max_rows_per_file * max_open_files. With this default, the max_open_rows would never be reached.

    This would help me in work I’m currently doing as I am currently setting a relatively low max_rows_per_file and max_open_files in order to limit memory usage, but this would allow me to set a dataset wide limit allowing individual files to potentially grow larger.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions