You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[C++] write_dataset: max_open_files does not close least recently used file #45038
For long running dataset writes with many partitions, this means that once max open files is saturated, over time the set of open files will trend towards being smaller and smaller in size. Recently opened files can begin to be closed prematurely, while old open files can hang around and take up a file slot.
It would probably make sense for C++ behaviour to be brought in line with the docs, which should be to close the least recently used file rather than the largest file, with the rationale that the least recently used file is more likely to be finished permanently.
A solution could be to store the last write time whenever DatasetWriterDirectoryQueue::StartWrite() is called (maybe someone more knowledgeable about the codebase can suggest somewhere better), and use that to replace DatasetWriter::TryCloseLargestFile() with something like DatasetWriter::TryCloseLeastRecentlyUsedFile().
changed the title [-]write_dataset max_open_files does not close oldest file[/-][+]write_dataset: max_open_files does not close least recently used file[/+]on Dec 17, 2024
It would probably make sense for C++ behaviour to be brought in line with the docs, which should be to close the least recently used file rather than the largest file, with the rationale that the least recently used file is more likely to be finished permanently.
Thanks for the report, and yes, I agree this would be a good idea.
Hmm, the current behavior seems to be deliberate according to the PR description:
FileSystemDatasetWriteOptions now has a max_open_files setting (ARROW-12321) which prevents opening too many files. Instead the writer will apply backpressure on the scanner while also closing the open file with the greatest # of rows already written (then resume writing once the file is closed).
@westonpace Do you happen to remember why the option description doesn't match the intended behavior? Is it just a mistake in the docstring?
changed the title [-]write_dataset: max_open_files does not close least recently used file[/-][+][C++] write_dataset: max_open_files does not close least recently used file[/+]on Dec 17, 2024
why the option description doesn't match the intended behavior? Is it just a mistake in the docstring?
I guess maybe using same strategy when memory overlimit and fd over limit? Since when we want to reclaim memory, close the largest file ( or k largest file) is usally a good way?
Before working on anything, we should decide which behavior is most desirable. If you want your dataset files to have relatively similar sizes, then it makes sense to close the largest files first.
There is a secondary write_dataset parameter in max_rows_per_file that can be used to control how large each file will get.
With the current implementation of max_open_files and writing a dataset with many partitions, rather than seeing file sizes remaining similar, you'd actually see file sizes steadily trend downwards due to small partition files that remain open, while newer partition files compete to stay under the file size of those smaller partitions.
With the current implementation of max_open_files and writing a dataset with many partitions, rather than seeing file sizes remaining similar, you'd actually see file sizes steadily trend downwards due to small partition files that remain open, while newer partition files compete to stay under the file size of those smaller partitions.
Ah, that may be the case if you have many partitions (more than max_open_files) that are very rarely hit?
yeah exactly, and I believe this would be a very common usecase, i.e. if you were to process/write a large file where the partition column is sorted (take a date partitioned file for example), new partitions would constantly be created while old partitions would never be touched again
As @mapleFU pointed out, it seems like this one parameter (max_open_files) is being used for two purposes, keep total file descriptors under a limit and keep memory usage reasonable.
Perhaps a second parameter should be added: max_open_rows. This would control memory usage while max_open_files controls fd use.
If max_open_rows is reached, the largest file is written (which would drop the number of open rows and memory usage the most). If max_open_files is reached, the least recently used file is written. This gives people the option to tune the performance as they need.
max_open_rows could be defaulted to max_rows_per_file * max_open_files. With this default, the max_open_rows would never be reached.
This would help me in work I’m currently doing as I am currently setting a relatively low max_rows_per_file and max_open_files in order to limit memory usage, but this would allow me to set a dataset wide limit allowing individual files to potentially grow larger.
Activity
[-]write_dataset max_open_files does not close oldest file[/-][+]write_dataset: max_open_files does not close least recently used file[/+]raulcd commentedon Dec 17, 2024
FYI @mapleFU @pitrou
pitrou commentedon Dec 17, 2024
Thanks for the report, and yes, I agree this would be a good idea.
pitrou commentedon Dec 17, 2024
Hmm, the current behavior seems to be deliberate according to the PR description:
@westonpace Do you happen to remember why the option description doesn't match the intended behavior? Is it just a mistake in the docstring?
mapleFU commentedon Dec 17, 2024
This idea LGTM, would you like to implement this? Or I'll check it then
[-]write_dataset: max_open_files does not close least recently used file[/-][+][C++] write_dataset: max_open_files does not close least recently used file[/+]xWaita commentedon Dec 17, 2024
Thanks for the replies! I can work on it
mapleFU commentedon Dec 17, 2024
I guess maybe using same strategy when memory overlimit and fd over limit? Since when we want to reclaim memory, close the largest file ( or k largest file) is usally a good way?
pitrou commentedon Dec 17, 2024
Before working on anything, we should decide which behavior is most desirable. If you want your dataset files to have relatively similar sizes, then it makes sense to close the largest files first.
xWaita commentedon Dec 17, 2024
There is a secondary write_dataset parameter in
max_rows_per_filethat can be used to control how large each file will get.With the current implementation of
max_open_filesand writing a dataset with many partitions, rather than seeing file sizes remaining similar, you'd actually see file sizes steadily trend downwards due to small partition files that remain open, while newer partition files compete to stay under the file size of those smaller partitions.pitrou commentedon Dec 17, 2024
Ah, that may be the case if you have many partitions (more than
max_open_files) that are very rarely hit?xWaita commentedon Dec 18, 2024
yeah exactly, and I believe this would be a very common usecase, i.e. if you were to process/write a large file where the partition column is sorted (take a date partitioned file for example), new partitions would constantly be created while old partitions would never be touched again
GH-44579: [C++] Use array type to compute min/max statistics Arrow ty…
ryancasburn-KAI commentedon Mar 22, 2025
As @mapleFU pointed out, it seems like this one parameter (
max_open_files) is being used for two purposes, keep total file descriptors under a limit and keep memory usage reasonable.Perhaps a second parameter should be added:
max_open_rows. This would control memory usage whilemax_open_filescontrols fd use.If
max_open_rowsis reached, the largest file is written (which would drop the number of open rows and memory usage the most). Ifmax_open_filesis reached, the least recently used file is written. This gives people the option to tune the performance as they need.max_open_rowscould be defaulted tomax_rows_per_file * max_open_files. With this default, themax_open_rowswould never be reached.This would help me in work I’m currently doing as I am currently setting a relatively low max_rows_per_file and max_open_files in order to limit memory usage, but this would allow me to set a dataset wide limit allowing individual files to potentially grow larger.