Skip to content

GzipFile.seek makes invalid write if buffer is not flushed in Python 3.12rc1 #108111

Description

@effigies

Bug report

Checklist

  • I am confident this is a bug in CPython, not a bug in a third-party project
  • I have searched the CPython issue tracker,
    and am confident this bug has not been reported before

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.12.0rc1 (main, Aug 16 2023, 05:03:59) [GCC 12.2.0]

A clear and concise description of the bug:

I have code that writes out sections of a data file in chunks, and uses seeks to ensure that the position is correct before writing.

In the following example, I write 5 bytes, seek to position 5 and write five more bytes. If I flush the buffer, the result is as expected. If I do not, 5 null bytes are written between the two groups of intended bytes.

#!/usr/bin/env python

import io
import gzip


for flush in (True, False):
    data = io.BytesIO()
    gzip_writer = gzip.GzipFile(fileobj=data, mode='wb')
    gzip_writer.write(b'abcde')

    # If the buffer isn't flushed, seek works from unchanged offset
    if flush and hasattr(gzip_writer, '_buffer'):
        gzip_writer._buffer.flush()

    gzip_writer.seek(5)
    gzip_writer.write(b'fghij')
    gzip_writer.close()
    
    # Recover result
    data.seek(0)
    gzip_reader = gzip.GzipFile(fileobj=data, mode='rb')
    result = gzip_reader.read()

    print(f'{flush=}: {result}')

In the case where I seek but don't tell, I get spurious \x00 bytes:

flush=True: b'abcdefghij'
flush=False: b'abcde\x00\x00\x00\x00\x00fghij'

Here is the output in Python 3.10.10:

flush=True: b'abcdefghij'
flush=False: b'abcdefghij'

Linked PRs

Activity

  1. effigies commented on Aug 18, 2023

    @effigies
    ContributorAuthor

    Looking at #101251, which is where I expect this was introduced, I suspect this is the fix:

         def seek(self, offset, whence=io.SEEK_SET):
             if self.mode == WRITE:
    +            self._check_not_closed()
    +            self._buffer.flush()
                 if whence != io.SEEK_SET:
                     if whence == io.SEEK_CUR:

    Here:

    cpython/Lib/gzip.py

    Lines 402 to 420 in daed54d

    def seek(self, offset, whence=io.SEEK_SET):
    if self.mode == WRITE:
    if whence != io.SEEK_SET:
    if whence == io.SEEK_CUR:
    offset = self.offset + offset
    else:
    raise ValueError('Seek from end not supported')
    if offset < self.offset:
    raise OSError('Negative seek in write mode')
    count = offset - self.offset
    chunk = b'\0' * self._buffer_size
    for i in range(count // self._buffer_size):
    self.write(chunk)
    self.write(b'\0' * (count % self._buffer_size))
    elif self.mode == READ:
    self._check_not_closed()
    return self._buffer.seek(offset, whence)
    return self.offset

  2. changed the title [-]Seeking to current position in writable gzip stream incorrectly appends zero bytes[/-] [+]GzipFile.seek makes invalid write if buffer is not flushed in Python 3.12rc1[/+] on Aug 22, 2023
  3. added a commit that references this issue on Aug 24, 2023
  4. added a commit that references this issue on Aug 24, 2023
  5. added a commit that references this issue on Aug 24, 2023
  6. hugovk commented on Nov 9, 2023

    @hugovk
    Member

    Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions