Repository navigation
C API: Detect buffer overflow in PyBytesWriter #156939
Description
PyBytesWriter gives a directy access to its content with PyBytesWriter_GetData(). The caller is responsible to handle correctly the buffer size and write the correct amount of bytes. But programming is hard, and mistakes are easy to do. I propose adding a mechanism to detect buffer overflow to help developers detecting bugs in in their code.
Example of bug:
PyBytesWriter *writer = PyBytesWriter_Create(0);
if (writer == NULL) {
goto error;
}
if (PyBytesWriter_Grow(writer, 3) < 0) {
goto error;
}
char *data = PyBytesWriter_GetData(writer);
strcpy(data, "abc");
return PyBytesWriter_Finish(writer);This code allocates 3 bytes but writes 4 bytes because strcat() also writes a trailing NUL byte.
Linked PRs
- gh-156939: Detect buffer overflow in PyBytesWriter in debug mode #156943
- gh-156939: Fix struct.pack('0p', bytes) #157071
- gh-156939: Fix memory corruption check in fcntl #157103
- gh-156939: Fix xmlcharrefreplace() buffer overflow #157109
- [3.15] gh-156939: Fix struct.pack('0p', bytes) (GH-157071) #157129
- [3.14] gh-156939: Fix struct.pack('0p', bytes) (GH-157071) #157130
- [3.13] gh-156939: Fix struct.pack('0p', bytes) (GH-157071) #157131
- [3.15] gh-156939: Fix xmlcharrefreplace() buffer overflow (GH-157109) #157232
- [3.14] gh-156939: Fix xmlcharrefreplace() buffer overflow (GH-157109) #157233
- [3.13] gh-156939: Fix xmlcharrefreplace() buffer overflow (GH-157109) #157234
- gh-156939: Document that PyBytesObject ends with a NUL byte #157236
- gh-156939: Detect buffer overflow in bytearray #157382
- gh-156939: Update UTF-8 encoder for PyBytesWriter changes #157383
- gh-156939: Detect PyBytesWriter buffer overflow earlier #157385
- gh-156939: Clear newly allocated bytes in PyBytesWriter_Resize() #157455
- gh-156939: Detect buffer overflow in bytes and bytearray #157529
- [3.15] gh-156939: Document that PyBytesObject ends with a NUL byte (GH-157236) #157550
- [3.14] gh-156939: Document that PyBytesObject ends with a NUL byte (GH-157236) #157551
- [3.13] gh-156939: Document that PyBytesObject ends with a NUL byte (GH-157236) #157552
- gh-156939: Document that PyASCIIObject ends with null byte #157708
- gh-156939: Add tests on the PyUnicode C API #157798
- gh-156939: Fix Modules/_testcapi/bytes.c: remove double Py_DECREF #158400
- gh-156939: Fix two bugs in PyBytesWriter #158404
- [3.15] gh-156939: Fix two bugs in PyBytesWriter (#158404) #158409
- [3.14] gh-156939: Document that PyUnicodeObject ends with null character (GH-157708) #159530
- [3.15] gh-156939: Document that PyUnicodeObject ends with null character (GH-157708) #159531
- [3.15] gh-156939: Add tests on the PyUnicode C API #159533
- [3.14] gh-156939: Add tests on the PyUnicode C API (#159533) #159539
Activity
pythongh-156939: Detect buffer overflow in PyBytesWriter in debug mode
pythongh-156939: Detect buffer overflow in PyBytesWriter in debug mode
pythongh-156939: Detect buffer overflow in PyBytesWriter in debug mode
pythongh-156939: Fix xmlcharrefreplace()
pythongh-156939: Fix xmlcharrefreplace() buffer overflow
vstinner commentedon Sep 7, 2026
Oh, I found multiple functions which write an extra (NUL) byte (buffer overflow!) or read the byte after the end of the buffer (read uninitialized memory):
struct.pack('0p', bytes)writes a trailing NUL byte.str.encode('latin1', 'xmlcharrefreplace')writes a trailing NUL byte.fcntl.fcntl()andfcntl.ioctl()readsdata[len]: read one byte after the end of the buffer (and expect it to be NUL).These functions rely on the fact that in C,
bytesobjects always end with a trailing NUL byte:assert(PyBytes_AS_STRING(obj)[PyBytes_GET_SIZE(obj)] == 0);is true (and doesn't read uninitialized memory, it's well defined). This bytes property is not documented in bytes C API documentation.vstinner commentedon Sep 7, 2026
See issue gh-155702 in sqlite3 where a bytes object created by PyBytesWriter was modified, whereas bytes objects are immutable. In this case, it was worse since the bytes object was a singleton object.
See also issue gh-155742 "Treat bytes objects as immutable in C code".
gh-156939: Fix struct.pack('0p', bytes) (#157071)
[3.13] gh-156939: Fix struct.pack('0p', bytes) (GH-157071) (#157131)
[3.14] gh-156939: Fix struct.pack('0p', bytes) (GH-157071) (#157130)
pythongh-156939: Fix xmlcharrefreplace() buffer overflow
pythongh-156939: Fix xmlcharrefreplace() buffer overflow
36 remaining items
pythongh-156939: Document that PyASCIIObject ends with null byte
pythongh-156939: Document that PyASCIIObject ends with null byte
pythongh-156939: Add tests on the PyUnicode C API
gh-156939: Document that PyUnicodeObject ends with null character (#1…
pythongh-156939: Add tests on the PyUnicode C API
gh-156939: Add tests on the PyUnicode C API (#157798)
pythongh-156939: Fix two bugs in PyBytesWriter
gh-156939: Fix Modules/_testcapi/bytes.c: remove double Py_DECREF (#1…
gh-156939: Fix two bugs in PyBytesWriter (#158404)
[3.15] gh-156939: Fix two bugs in PyBytesWriter (#158404) (#158409)
[3.15] gh-156939: Fix struct.pack('0p', bytes) (GH-157071) (GH-157129)
[3.14] gh-156939: Document that PyUnicodeObject ends with null charac…
[3.15] gh-156939: Document that PyUnicodeObject ends with null charac…
[3.15] gh-156939: Add tests on the PyUnicode C API (#159533)
[3.14] gh-156939: Add tests on the PyUnicode C API (#159533) (#159539)