Repository navigation
TemporaryDirectory clean-up fails with unsearchable directories #79325
Description
Activity
lilydjwg commented
on Nov 2, 2018 lilydjwgmannequinMannequinAuthorIf the title doesn't explain clearly, here's a demo program that will fail:
import tempfile import pathlib def test(): with tempfile.TemporaryDirectory(prefix='test-bad-') as tmpdir: tmpdir = pathlib.Path(tmpdir) subdir = tmpdir / 'sub' subdir.mkdir() with open(subdir / 'file', 'w'): pass subdir.chmod(0o600) if __name__ == '__main__': test()
I didn't expect this, and I didn't find an easy way to handle this except not using TemporaryDirectory at all:
import tempfile import pathlib import shutil import os def rmtree_error(func, path, excinfo): if isinstance(excinfo[1], PermissionError): os.chmod(os.path.dirname(path), 0o700) os.unlink(path) print(func, path, excinfo) def test(): tmpdir = tempfile.mkdtemp(prefix='test-good-') try: tmpdir = pathlib.Path(tmpdir) subdir = tmpdir / 'sub' subdir.mkdir() with open(subdir / 'file', 'w'): pass subdir.chmod(0o600) finally: shutil.rmtree(tmpdir, onerror=rmtree_error) if __name__ == '__main__': test()
This works around the issue, but the dirfd is missing in the onerror callback.
I have this issue because my program extracts tarballs to a temporary directory for examination. I expected that TemporaryDirectory cleaned up things when it could.
What do you think? rm -rf can't remove such a directory either but this is annoying and I think Python can do better.
- changed the title
[-]TemporaryDirectory can't be cleaned up if there are unsearchable directories[/-][+]TemporaryDirectory clean-up fails with unsearchable directories[/+]on Nov 2, 2018 lilydjwg commented
on Nov 3, 2018 lilydjwgmannequinMannequinAuthorYes bpo-26660 is similar but not the same. On Windows it seems a read-only file cannot be deleted while on Linux a file resident in a non-searchable directory cannot be deleted.
An onerror for TemporaryDirectory will work. Also I'd like to add an optional dir_fd argument for onerror.
I am working on this. Left to test on Windows and analyze possible security issues.
vidartf commented
on Sep 22, 2020 vidartfmannequinMannequinOn Python 3.8.5 on Windows using the code from the above patch I recently got a stack overflow:
Thread 0x00002054 (most recent call first):
File "...\lib\concurrent\futures\thread.py", line 78 in _worker
File "...\lib\threading.py", line 870 in run
File "...\lib\threading.py", line 932 in _bootstrap_inner
File "...\lib\threading.py", line 890 in _bootstrapThread 0x00000de4 (most recent call first):
File "...\lib\concurrent\futures\thread.py", line 78 in _worker
File "...\lib\threading.py", line 870 in run
File "...\lib\threading.py", line 932 in _bootstrap_inner
File "...\lib\threading.py", line 890 in _bootstrapCurrent thread 0x00004700 (most recent call first):
File "...\lib\tempfile.py", line 803 in onerror
File "...\lib\shutil.py", line 619 in _rmtree_unsafe
File "...\lib\shutil.py", line 737 in rmtree
File "...\lib\tempfile.py", line 814 in _rmtree
File "...\lib\tempfile.py", line 806 in onerror
File "...\lib\shutil.py", line 619 in _rmtree_unsafe
File "...\lib\shutil.py", line 737 in rmtree
... repeating-------------------------------------------
In my case, the outer
exc_infofrom rmtree is:PermissionError(13, 'The process cannot access the file because it is being used by another process')
And the inner exception from
_os.unlink(path)is:PermissionError(13, 'Access is denied')
I would say that expected behavior in this case would be to let the 'file is in use' error raise, instead of killing the process with an SO.
Thank you Vidar! I wasn't sure about Windows, but was not able to reproduce possible failures. Your report gives a clue.
vidartf commented
on Sep 22, 2020 vidartfmannequinMannequinA somewhat easy repro:
Create the temporary directory, add a subdir (not sure if subdir truly necessary at this point), use
os.chdir()to set the cwd to that subdir. Clean up the temp dir. The cwd should prevent the deletion because it will be "in use".It seems to me that if
path == name, then resetperms(path) and possibly a recursive call are only needed on the first call. In subsequent calls, ifpath == name, then we know that resetperms(path) was already called, so it shouldn't handle PermissionError. If resetperms was ineffective (e.g. in Windows, a sharing violation or custom discretionary/mandatory permissions), or if something else changed the permissions in the mean time, just give up instead of risking a RecursionError or stack overflow. For example:@classmethod def _rmtree(cls, name, first_call=True): resetperms_funcs = (_os.unlink, _os.rmdir, _os.scandir, _os.open) def resetperms(path): try: _os.chflags(path, 0) except AttributeError: pass _os.chmod(path, 0o700) def onerror(func, path, exc_info): if (issubclass(exc_info[0], PermissionError) and func in resetperms_funcs and (first_call or path != name)): try: if path != name: resetperms(_os.path.dirname(path)) resetperms(path) try: _os.unlink(path) # PermissionError is raised on FreeBSD for directories except (IsADirectoryError, PermissionError): cls._rmtree(path, first_call=False) except FileNotFoundError: pass elif issubclass(exc_info[0], FileNotFoundError): pass else: raise _shutil.rmtree(name, onerror=onerror)
2 remaining items
- added a commit that references this issue
on Dec 1, 2022 I implemented this idea in #112762.
- added a commit that references this issue
on Dec 7, 2023 - added 3 commits that reference this issue
on Dec 7, 2023 - added a commit that references this issue
on Feb 11, 2024
Metadata
Metadata
Assignees
Labels
Projects
- Status
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs