Repository navigation
gzip raising exception when closing with buffer backed by BytesIO #129726
Description
Activity
- changed the title
[-]gzip raising exception when closing empty buffer backed by bytesio[/-][+]gzip raising exception when closing empty buffer backed by BytesIO[/+]on Feb 6, 2025 - changed the title
[-]gzip raising exception when closing empty buffer backed by BytesIO[/-][+]gzip raising exception when closing with buffer backed by BytesIO[/+]on Feb 7, 2025 Explicit
buffer.close()anddel bufferin the method make the error go away, so I think this has to do with ordering of cleanup of references when buffer is actually deallocated. Moving towith gzip.GzipFile ... as buffer:also doesn't error. The issue seems to be that theio.BytesIOis cleaned up before thegzip.GzipFile.The same behavior occurs with an explicit reference to the
io.BytesIO:def explicit_reference(): bio = io.BytesIO() buffer = gzip.GzipFile(fileobj=bio, mode="w")
Can work around it so no longer error at least by updating
GzipFile.close()to checkif self.fileobj.closedand fast-path exiting (See: https://github.com/python/cpython/compare/main...cmaloney:cpython:exp/gzip?expand=0). Not sure that is the right approach though vs. getting the objects to destruct in the right order.Explicit
buffer.close()anddel bufferin the method make the error go away, so I think this has to do with ordering of cleanup of references when buffer is actually deallocated. Moving towith gzip.GzipFile ... as buffer:also doesn't error.Thanks for testing these, I cannot use the context manager but I may be able to close the buffer explictly.
The issue seems to be that the
io.BytesIOis cleaned up before thegzip.GzipFile.Yeah, that is my impression too
The same behavior occurs with an explicit reference to the
io.BytesIO:def explicit_reference():
bio = io.BytesIO()
buffer = gzip.GzipFile(fileobj=bio, mode="w")Can work around it so no longer error at least by updating
GzipFile.close()to checkif self.fileobj.closedand fast-path exiting (See: https://github.com/python/cpython/compare/main...cmaloney:cpython:exp/gzip?expand=0). Not sure that is the right approach though vs. getting the objects to destruct in the right order.Yeah would be nice to keep the code working instead of not just crashing :) The gzip code should already have references to the fileobj so to me is surprising it gets deallocated while we are in in GzipFile.close.
Reacted by Cody MaloneySo my original code when getting the gzipped content is this so am not sure I can workaround this other than stop using GzipFile.
fileobj = buffer.fileobj # get a reference to the fileobj before closing the gzip file buffer.close() data = fileobj.getbuffer()Irony of the comment 😅
The documentation explicitly says that calling
GzipFile'sclose()does not close the fileobj, so I guess it also means it should not be freed:Calling a GzipFile object’s close() method does not close fileobj, since you might wish to append more material after the compressed data. This also allows you to pass an io.BytesIO object opened for writing as fileobj, and retrieve the resulting memory buffer using the io.BytesIO object’s getvalue() method.I still haven't found what is causing the fileobj to be closed... The line which is causing the exception print was added to 3.12 in https://github.com/python/cpython/pull/105920/files it looks like to fix #105808
@cmaloney the one you linked touches
flush, here is crashing inclose. AFAICS the only commit that touchesclose(and coincidentally the gc 😅 ) that is in 3.13 but not in 3.12 is b52fc70#diff-ad9b54ac8ef847cbb11fb0550f7e8cede55b1d92de15899e0a885a94a124838aI think it's the move to using a BufferedWriter (gh-89550) changed the behavior around flush and close in some unintended ways (https://github.com/python/cpython/blob/main/Modules/_io/bufferedio.c#L543-L598 being one of the parts). That the stream is being marked as "closed", but later gets flushed to. Still investigating / don't have any verifiable answers yet, just hunches.
Reacted by Riccardo MagliocchettiOk so with 3.12 and development mode I see the same warning, don't see the warning in 3.10 and 3.11 though:
$ python3.13 gzipfoo.py Exception ignored in: <gzip on 0x7f39c7098580> Traceback (most recent call last): File "/usr/lib/python3.13/gzip.py", line 359, in close fileobj.write(self.compress.flush()) ValueError: I/O operation on closed file. $ python3.12 gzipfoo.py $ python3.12 -X dev gzipfoo.py Exception ignored in: <gzip on 0x7f1450669590> Traceback (most recent call last): File "/usr/lib/python3.12/gzip.py", line 357, in close fileobj.write(self.compress.flush()) ValueError: I/O operation on closed file. $ /usr/bin/python3.10 -X dev gzipfoo.py $ /usr/bin/python3.11 -X dev gzipfoo.pySo there is something that changed the behavior in 3.12.
4 remaining items
Thanks a lot!
Reacted by Cody MaloneyMaybe GzipFile should emit a
ResourceWarningif the file is not closed explicitly to make sure that data are written on disk.Reacted by Cody MaloneyI implemented a ResourceWarning on
BufferedWriterchecking for unwritten data, and just the CPython I/O tests it fires a lot. Even with conditions (only if raw isn't closed, etc) many cases to work through. I think worthwhile to help track down and cleanup inconsistencies at least in CPython itself. Not sure if it's going to be too noisy in general Python ecosystem codebases though.The test suite can be fixed. The risk of losing data justify emitting ResourceWarning.
Reacted by Cody Maloney- added a commit that references this issue
on Mar 6, 2025 - added a commit that references this issue
on Mar 8, 2025 Hey @xrmx I see that a new CPython version was released, looks like it does include the fix of the issue you reported: https://github.com/python/cpython/releases/tag/v3.13.3
Bug report
Bug description:
Hello,
the following snippet raises an exception in Python 3.13.2 while it's fine with Python 3.12.
Running this with python3.12 is silent, with Python 3.13.2 instead:
CPython versions tested on:
3.13
Operating systems tested on:
Linux
Linked PRs
gzip.GzipFilereference loop #130055gzip.GzipFilereference loop (GH-130055) #130669gzip.GzipFilereference loop (GH-130055) #130670