Repository navigation
glob empty return when ulimit -n is reached #103501
Description
Activity
- changed the title
[-]`glob` empty return when `ulimit` is reached[/-][+]`glob` empty return when `ulimit -n` is reached[/+]on Apr 13, 2023 - added a commit that references this issue
on Oct 19, 2023 - added a commit that references this issue
on Nov 6, 2023 - added a commit that references this issue
on Jan 11, 2024 What is the behavior of the pathname expansion in the Unix shell in such case? Errors like broken symlinks, permission deny, too long path are ignored. Maybe all errors are ignored. And
glob.glob()conforms this.Maybe in future we will add an option to make
glob.glob()not ignoring all OS errors, or even specify an error handler to ignore only some errors.Maybe in future we will add an option to make glob.glob() not ignoring all OS errors
This might be a good use case for ExceptionGroup :)
The current behaviour is that OSError are ignored. See recent discussions in e.g. #104292 and (for
pathlib's glob) 104141.But
EMFILEis a bit different than permission issues or disk failures: it's transient, it'll go away if you close a few files. Perhaps it is worth it to treat it specially.
@barneygale, do you have an opinion here? (I assumepathlib's behaviour should matchglob's)It is not a good case for ExceptionGroup. There might be many thousands files and directories in the tree.
There are three strategies to handle errors:
- Fail. Raise an exception to signal about error and stop the whole process.
- Ignore. In the best case the errors can be logged or saved for further analysis.
- Try to fix and repeat the failed operation.
The "fail" and "ignore" strategies are supported in other complex operations (like
os.walk(),shutil.rmtree(),shutil.copytree()) and controlled by specifying a user callbacks or just a boolean flag. But the "retry" strategy is more complex and is not well supported in the current stdlib code. For example, to fix EMFILE inglob()the user needs to close some other file descriptors, theglob()code cannot know what file descriptors can be closed and when they are closed. And then there is a question what part of the code should be repeated after the fix.FWIW, pathlib calls
stat()on the top-level path and suppresses some (but not all)OSErrors. Any errors from deeper paths are totally suppressed.Reacted by Petr ViktorinRight. ExceptionGroup only works for “ignore” (for logging the errors).
The 3 strategies can be mixed. Thinking about it, in most of my uses of glob, I'm OK with ignoring
PermissionError(not listing inaccessible files), but would rather fail onEMFILE(where I'm losing info about files that are normally accessible).
And that's what's reported in this issue. The trouble is that if we go that way, we'd be expected to have a good default for all errors on all platforms, which probably isn't feasible. The user callback seems better -- even if it doesn't support retrying.- added a commit that references this issue
on Sep 17, 2025 - added a commit that references this issue
on Apr 14, 2026
Bug report
When having a lot of opened files, glob can return an empty list rather than either raising an error or returning a non-empty list.
With a
ulimitset at 256 (ulimit -n 256), and with a file namedfoo:My intuition is that
globshould inform the user that it couldn't do its job properly (likeopenraisesOSErrorif we try to open too many files), because right now there's an uncertainty on whether a given folder is empty orulimit -nhas been reached.An OSError is caught in
glob.py, but not propagated: https://github.com/python/cpython/blob/3.11/Lib/glob.py#L172Your environment
python:latest), both x86_64Linked PRs