subprocess on Windows returns None instead of raising UnicodeDecodeError #158006

Bug description:

On Windows, Popen.communicate() reads stdout and stderr in background threads when a timeout is given, or when more than one standard stream is a pipe. In text mode, the decoding also happens in these threads, in _readerthread():

If decoding fails, the UnicodeDecodeError stays in the thread. It is only printed as "Exception in thread ... (_readerthread)". The caller never gets it, and communicate() returns None for that stream:

So subprocess.run(..., capture_output=True, text=True) returns normally with stdout=None. The program then fails later with a confusing error, like TypeError: ... 'NoneType'.

The other code paths raise the error. On POSIX, _communicate() decodes in the calling thread. On Windows with one pipe and no timeout, communicate() also reads in the calling thread:

Minimal example:

import subprocess, sys

# The child process writes bytes that are not valid UTF-8.
cmd = [sys.executable, "-c", "import sys; sys.stdout.buffer.write(b'ok \\xff\\n')"]

def run(label, **kwargs):
    try:
        r = subprocess.run(cmd, encoding="utf-8", **kwargs)
        print(label, "-> returned, stdout =", repr(r.stdout), flush=True)
    except UnicodeDecodeError as e:
        print(label, "-> raised", repr(e), flush=True)

run("stdout=PIPE", stdout=subprocess.PIPE)
run("stdout=PIPE, timeout=10", stdout=subprocess.PIPE, timeout=10)
run("capture_output=True", capture_output=True)

Output on Windows 11, Python 3.14.5:

stdout=PIPE -> raised UnicodeDecodeError('utf-8', b'ok \xff\n', 3, 4, 'invalid start byte')
Exception in thread Thread-1 (_readerthread):
Traceback (most recent call last):
  File "...\Lib\threading.py", line 1082, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "...\Lib\threading.py", line 1024, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Lib\subprocess.py", line 1614, in _readerthread
    buffer.append(fh.read())
                  ~~~~~~~^^
  File "<frozen codecs>", line 325, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 3: invalid start byte
stdout=PIPE, timeout=10 -> returned, stdout = None
Exception in thread Thread-2 (_readerthread):
Traceback (most recent call last):
  File "...\Lib\threading.py", line 1082, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "...\Lib\threading.py", line 1024, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Lib\subprocess.py", line 1614, in _readerthread
    buffer.append(fh.read())
                  ~~~~~~~^^
  File "<frozen codecs>", line 325, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 3: invalid start byte
capture_output=True -> returned, stdout = None

3.12.13, 3.13.13 and 3.15.0b1 give the same three results.

Output on Linux (Ubuntu on WSL 2), Python 3.12.3:

stdout=PIPE -> raised UnicodeDecodeError('utf-8', b'ok \xff\n', 3, 4, 'invalid start byte')
stdout=PIPE, timeout=10 -> raised UnicodeDecodeError('utf-8', b'ok \xff\n', 3, 4, 'invalid start byte')
capture_output=True -> raised UnicodeDecodeError('utf-8', b'ok \xff\n', 3, 4, 'invalid start byte')

I expected all three calls to raise UnicodeDecodeError on Windows too, like on Linux.

This will happen more often with 3.15. Python 3.15 uses UTF-8 as the default encoding (What's New), so text=True now decodes as UTF-8 on Windows. But many Windows programs still write in the console code page. For example, on my Traditional Chinese Windows (code page 950):

import subprocess
r = subprocess.run(["where", "no_such_program"], capture_output=True, text=True)
print("returncode =", r.returncode, "stderr =", repr(r.stderr))

Python 3.14.5:

returncode = 1 stderr = '資訊: 找不到提供模式的檔案。\n'

Python 3.15.0b1:

Exception in thread Thread-2 (_readerthread):
Traceback (most recent call last):
  File "...\Lib\threading.py", line 1218, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "...\Lib\threading.py", line 1160, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Lib\subprocess.py", line 1774, in _readerthread
    buffer.append(fh.read())
                  ~~~~~~~^^
  File "<frozen codecs>", line 325, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb8 in position 0: invalid start byte
returncode = 1 stderr = None

With a UnicodeDecodeError, the user would know that they need encoding= or errors=. With None, they don't.

This has been seen before:

• Subprocess IndexError possible in _communicate #87589 / bpo-43423 Fix IndexError in subprocess _communicate function #24777 (2021) fixed an IndexError on the same lines by returning None. In the review, it was noted that this only happens when a reader thread fails (bpo-43423 Fix IndexError in subprocess _communicate function #24777 (comment)). A decode error in text mode is an easy way for the reader thread to fail.
• When installing certain libraries using pip in Python 3.15.0a2, pip reports an error #141877: a user of 3.15.0a2 saw these _readerthread tracebacks when pip built numpy. It was closed for another reason.
• In the PR that made UTF-8 mode the default (GH-133711: Enable UTF-8 mode by default (PEP 686) #133712), a Windows buildbot failed in test_regrtest with TypeError: expected string or bytes-like object, got 'NoneType' after the same thread traceback (GH-133711: Enable UTF-8 mode by default (PEP 686) #133712 (comment)). The test was fixed with errors="backslashreplace" in gh-133711: Fix test_regrtest for PYTHONUTF8=1 #134839.

The open PR GH-155409 (for gh-87512) changes the reader threads to read bytes, and decodes in _communicate() like the POSIX code. I ran my example with Lib/subprocess.py from that PR (without its C changes, which this path does not use), and all three calls raised UnicodeDecodeError. So that PR would also fix this on main. But it is a bigger change, and it has no test for this case.

A smaller fix: in _readerthread(), catch the exception and put it in the buffer. Then _communicate() re-raises it in the calling thread after the threads are joined. I tried this change (about 10 lines) on a copy of Lib/subprocess.py from main, with 3.14.5 and 3.15.0b1. With it, all three calls in the example raise UnicodeDecodeError, and bytes mode and the errors= handlers work as before.

I'd be happy to open a PR with this fix and a test, if this approach is acceptable. If GH-155409 is the preferred way, I could send a test for this case instead.

English is not my first language. I used an AI assistant to help write this issue and the reproduction script. The outputs above are from real runs on my machine.

CPython versions tested on:

3.12, 3.13, 3.14, 3.15

Operating systems tested on:

Windows