Repository navigation
Unfixable potential race conditions on file descriptors in CPython #121544
Description
Python allows multiple threads to concurrently access file descriptors through files and sockets. Race conditions at the Python level can lead to unexpected behavior, even with the global interpreter lock.
Thread sanitizer reports these races in some of our tests that exercise this behavior. We cannot fix these potential race conditions without introducing potential deadlocks.
For example, consider:
import threading
secrets = open("secrets.txt", "w");
def thread1():
secrets.write("a secret")
def thread2():
secrets.close()
def thread3():
with open("log.txt", "w") as log:
log.write("log message")
threading.Thread(target=thread1).start()
threading.Thread(target=thread2).start()
threading.Thread(target=thread3).start()If you are particularly unlucky, then the file descriptor for secrets may be closed by thread2 and reused as the file descriptor for log just before thread1 writes to it. In other words, thread1 may write "a secret" to a completely different file or socket than it intended.
This can happen, even with the GIL, because the GIL is released around write() and close(). In other words, the following can happen:
- The
secrets.write()call releases the GIL, but before it actually calls the Cwrite()function on the file descriptor.... - The
secrets.close()call closes the file descriptor - The
open("log.txt", "w")re-uses the same file descriptor number - The
secrets.write()call nowwrite()on the wrong file descriptor
Note that we must release the GIL before calling write() or close() because these functions potentially block.
Activity
[-]Unfixable potential race conditions on file descriptor in CPython[/-][+]Unfixable potential race conditions on file descriptors in CPython[/+]serhiy-storchaka commentedon Jul 10, 2024
What concrete parts of the stdlib or tests have this race condition?
There were some issues in multiprocessing, I tried to avoid introducing such kind of race condition.
pythongh-121592: Make select.poll() and related objects thread-safe
pythongh-121592: Make select.poll() and related objects thread-safe
gh-121592: Make select.poll() and related objects thread-safe (#121594)
pythongh-121592: Make select.poll() and related objects thread-safe (p…
[3.13] gh-121592: Make select.poll() and related objects thread-safe (G…
pythongh-121592: Make select.poll() and related objects thread-safe (p…
colesbury commentedon Jul 11, 2024
@serhiy-storchaka I don't think the stdlib has internal races on file descriptors. We have some in tests, but it may take me a bit to find them because we still have other race conditions reported by TSan, and it's easy to get them mixed up.
test_socket.testClose- race betweenrecv()andconnect()reported by TSan. This seems benign to me -- unlike my original example, neither call creates or closes file descriptors -- but I'm not entirely sure.pythongh-121592: Make select.poll() and related objects thread-safe (p…
colesbury commentedon Mar 21, 2025
@kumaraditya303 fixed the file descriptor race in the asyncio tests:
I'm going to close this issue as I don't think there's anything actionable left to do.