Repository navigation
subprocess.Popen.communicate does not send input #141473
Description
Bug report
Bug description:
I will start with saying that documentation for subprocess.Popen is not clear. I was using subprocess.run(..., input=..., ...) and needed to monitor other things while the created process is running, so I changed it to subprocess.Popen. The only way to send input was Popen.communicate(input=...), however I wanted to have a timeout. My initial idea was .communicate(input=..., timeout=0), catch the exception, and then continue with .wait(timeout=STEP), however it was hanging. I worked it around by launching .communicate without timeout in a separate thread.
Then, I tested a few platforms and looked at the implementation. On Windows, my initial idea works. On Linux, it hangs. After looking into the implementation, I deduced I shouldn't use .wait(), but rather .communicate() and set the input only for the first invocation. If that's the intention, it would be great to have it documented.
Anyway, the following example hangs on a few Linux versions and few Python versions, including current main branch:
import subprocess
proc = subprocess.Popen(
["cat", "-"],
stdin=subprocess.PIPE
)
try:
proc.communicate(input="abcd\n".encode(), timeout=0)
except subprocess.TimeoutExpired:
pass
print("it will hang:")
proc.communicate() # initially: proc.wait()
print("finished")It looks like a bug, and this line:
Line 2108 in 9cd5427
which should be changed to:
if self.stdin and not self.stdin.closed and self._input:After this change, the example above no longer hangs. I'll open PR with the mentioned change.
CPython versions tested on:
3.12, 3.14, CPython main branch
Operating systems tested on:
Linux, Windows
Linked PRs
- gh-141473: Fix subprocess.Popen.communicate to send input to stdin upon a subsequent post-timeout call #141477
- [3.14] gh-141473: Fix subprocess.Popen.communicate to send input to stdin upon a subsequent post-timeout call (GH-141477) #142059
- [3.13] gh-141473: Fix subprocess.Popen.communicate to send input to stdin upon a subsequent post-timeout call (GH-141477) #142060
- gh-141473: Document not calling Popen.wait after Popen.communicate times out. #142101
- [3.14] gh-141473: Document not calling Popen.wait after Popen.communicate times out. (GH-142101) #142124
- [3.13] gh-141473: Document not calling Popen.wait after Popen.communicate times out. (GH-142101) #142125
- gh-141473: Speed up subprocess test_communicate_timeout_large_input long tail #149003
- [3.14] gh-141473: Speed up subprocess test_communicate_timeout_large_input long tail (GH-149003) #149004
- [3.13] gh-141473: Speed up subprocess test_communicate_timeout_large_input long tail (GH-149003) #149005
Activity
pythongh-141473: Fix subprocess.Popen.communicate to send input to stdin
gpshead commentedon Nov 27, 2025
Good catch and reproducer. As always within the subprocess state machine I had to read over a lot to convince myself of its behavior and what was supported. Your proposed fix PR is simple and safe from what I can tell. I've also updated the docs in it to mention the caveat about input being a one-time only parameter.
I think what happened over time here is a mix:
Popen.communicatewas not originally intended to be called more than once as there had been no reason to do so - the process was supposed to have died by the time it returned. But the 3.3 addition of timeout support provided an actual use case including internally keeping track of the original input buffer and continuing to write what was left of that to the pipe on later post-timeout communicate() calls.Popen does not expose any intentional public API to indicate of how much of the communicate() input= has been written to the stdin pipe after a timeout exception. Later calls to communicate if they also supply an input parameter per my read of the code today simply have that new input ignored. I'm documenting that as explicitly undefined behavior for now as part of this PR. We could choose to change that to raise or to try and accumulate more input into the internal buffer in the future but that'd need to be a deliberate feature. I doubt we should do so (complexity).
Either way, this is somewhat of a niche use case as once you want to let a process run on its own with a timeout with pipe(s) of data in and or out of it all at once, you either need to be using threads or async / selectors to avoid anything that blocks. Stepping on your own via repeated timeouts can artificially limit how rapidly the pipe data is written to the process more than would otherwise be natural.
gh-141473: Fix subprocess.Popen.communicate to send input to stdin up…
pythongh-141473: Fix subprocess.Popen.communicate to send input to st…
pythongh-141473: Fix subprocess.Popen.communicate to send input to st…
Add regression test for subprocess timeout during large input write
Add regression test for subprocess timeout during large input write
Add regression test for subprocess timeout during large input write
pythongh-141473: Fix subprocess communicate() timeout on Windows for …
[3.14] gh-141473: Fix subprocess.Popen.communicate to send input to s…
[3.13] gh-141473: Fix subprocess.Popen.communicate to send input to s…
3 remaining items
pythongh-141473: Document not calling Popen.wait after Popen.communic…
pythongh-141473: Document not calling Popen.wait after Popen.communic…
[3.14] gh-141473: Document not calling Popen.wait after Popen.communi…
[3.13] gh-141473: Document not calling Popen.wait after Popen.communi…
pythongh-141473: Document not calling Popen.wait after Popen.communic…
mrowqa commentedon Dec 2, 2025
Thanks for the further context on the "undefined" behavior.
Yes - it looks great! Thank you.
pythongh-141473: Fix subprocess.Popen.communicate to send input to st…
pythongh-141473: Document not calling Popen.wait after Popen.communic…
gh-141473: Speed up subprocess test_communicate_timeout_large_input l…
[3.13] gh-141473: Speed up subprocess test_communicate_timeout_large_…
[3.14] gh-141473: Speed up subprocess test_communicate_timeout_large_…