Repository navigation
BlockingIOError: [Errno 11] Resource temporarily unavailable: on GPFS. #87909
Description
| BPO | 43743 |
|---|---|
| Nosy | @gpshead, @giampaolo, @alexeicolin, @pmrv |
| PRs | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-05-06.19:04:38.663>
created_at = <Date 2021-04-06.08:21:31.814>
labels = ['type-bug', '3.8', '3.9', '3.10', 'expert-IO', 'invalid']
title = 'BlockingIOError: [Errno 11] Resource temporarily unavailable: on GPFS.'
updated_at = <Date 2022-03-13.15:22:25.341>
user = 'https://bugs.python.org/pconesamingo'bugs.python.org fields:
activity = <Date 2022-03-13.15:22:25.341>
actor = 'pmrv'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-06.19:04:38.663>
closer = 'gregory.p.smith'
components = ['IO']
creation = <Date 2021-04-06.08:21:31.814>
creator = 'p.conesa.mingo'
dependencies = []
files = ['50671', '50672']
hgrepos = []
issue_num = 43743
keywords = ['patch']
message_count = 9.0
messages = ['390297', '391649', '392985', '393134', '393364', '393419', '393429', '415037', '415038']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'giampaolo.rodola', 'alexeicolin', 'p.conesa.mingo', 'PEAR', 'pmrv']
pr_nums = ['26024']
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43743'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
Activity
pconesamingo commentedon Apr 6, 2021
Hi, one of our users is reporting this starting to happen in a GPFS. All has been working fine for NTFS so far for many years.
I had a look at my shutil code, and I can see the try/except code trying to fall back to the "slower" copyfileobj(fsrc, fdst).
But it seems, by the stacktrace bellow that the "catch" is not happening.
Any idea how to fix this?
I guess something like:
should avoid the fast_copy attempt.
alexeicolin commentedon Apr 23, 2021
Can confirm that this BlockingIOError happens on GPFS (alpine) on Summit supercomputer, tested with Python 3.8 and 3.10a7.
I found that it happens only for file sizes above 65536. Minimal example:
This filesize works:
This file size (and larger) does not work:
I tried patching shutil.py to retry the the call on this EAGAIN, but subsequent attempts fail with EAGAIN again indefinitely.
I also use OP's workaround: set _USE_CP_SENDFILE = False in shutil.py
PEAR commentedon May 5, 2021
Most probably related: https://www.ibm.com/support/pages/apar/IJ28891
gpshead commentedon May 6, 2021
I don't believe CPython should be working around a bug in specific Linux kernel versions in the standard library unless they are extremely pernicious and not considered to be a bug and thus ever be fixed in the OS kernel.
As the sendfile system call appears to infinitely return one of EAGAIN, EALREADY, EWOULDBLOCK, or EINPROGRESS in this case, there isn't anything CPython could do. A retry/backoff loop won't help.
This should be worked around at the application level by whatever means are appropriate.
7 remaining items
pconesamingo commentedon May 10, 2021
So, is it ok, when the fast copy fails, not to _GiveupOnFastCopy(err)?
I can understand that fast copy might fail, but then the Giveup part should happen and it wasn't.
Additionally, _USE_CP_SENDFILE could be taken, optionally from an environment variable to cancel the fastcopy once we know it will fail?
gpshead commentedon May 10, 2021
The logic for bailing out to a slow copy is currently:
https://github.com/python/cpython/blob/main/Lib/shutil.py#L158
that condition appears to not be happening in Alexei's test. Suggesting that either at least one sendfile call succeeded and thus offset is non-zero or the lseek failed.
run that test under pdb and walk thru the code, or under strace to look at the syscalls and find out.
The question seems to be is if it should be okay to _GiveUpOnFastCopy after a partial (incomplete) copy has already occurred via sendfile.
giampaolo commentedon May 10, 2021
I think it should not. For posterity: my rationale for introducing _USE_CP_SENDFILE was to allow monkey patching for corner cases such as this one (see also bpo-36610 / #57884), but expose it as a private name because I expected them to be rare and likely up to a broken underlying implementation, as it appears this is the case. FWIW, I deem _USE_CP_SENDFILE usage in production code as legitimate, and as such it should stay private but never be removed.
pmrv commentedon Mar 13, 2022
I hope you don't mind me necro posting, but I ran into this issue again and
have a small patch to solve it.
I attached an MWE that triggers the BlockingIOError reliably on ext4
filesystems in linux 4.12.14 and python 3.8.12. Running under strace -e
sendfile gives the following output
# manually calling sendfile to check that it works
This shows that the first call to sendfile actually copies the whole file and
the EAGAIN is only triggered on the second, unnecessary, call. I have tested
with a small C program that it's triggered whenever sendfile's offset + count
exceeds the file size of in_fd. This is weird behaviour on the kernels side
that seems to have changed in newer kernel versions (issue is not present e.g.
on my 5.16.12 laptop).
Anyways my patch makes that second call not appear by keeping track of the file
size and the bytes written so far. It's against the current python main
branch, but if I see correctly this part hasn't changed in years. I have
checked the error is not thrown when the patch is applied.
(I can only attach one file, so patch is attached in a new one.)
pmrv commentedon Mar 13, 2022
Here's the small patch. Sadly I have no overview what the affected linux kernel version are. I guess technically you can all this "working around a bug in specific linux version", but since it's a very minor change that saves one syscall even for non-breaking version, I feel it's justified. Let me know if you'd like any modification done however.
jmozmoz commentedon Jan 21, 2023
I have exectly the same problem (using jupyter lab) and the patch provided solves the problem.
pythongh-87909: avoid a 0 bytes copy call to sendfile()
gpshead commentedon Jan 22, 2023
said patch leads to other test failures based on the linux CI on that PR.
jmozmoz commentedon Jan 22, 2023
Just for reference here the used version: