GzipFile.write should be buffered

@marcelm

BPO	45387
Nosy	@marcelm, @rhpvorderman

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-10-06.08:42:34.497>
labels = ['3.8', '3.9', '3.10', 'performance', '3.7', 'library', '3.11']
title = 'GzipFile.write should be buffered'
updated_at = <Date 2021-10-06.08:59:49.938>
user = 'https://github.com/rhpvorderman'

bugs.python.org fields:

activity = <Date 2021-10-06.08:59:49.938>
actor = 'marcelm'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-10-06.08:42:34.497>
creator = 'rhpvorderman'
dependencies = []
files = []
hgrepos = []
issue_num = 45387
keywords = []
message_count = 1.0
messages = ['403289']
nosy_count = 2.0
nosy_names = ['marcelm', 'rhpvorderman']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue45387'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

gh-89550: Buffer GzipFile.write to reduce execution time by ~15% #101251

Please consider the following code snippet:

    import gzip
    import sys

    with gzip.open(sys.argv[1], "rt") as in_file_h:
        with gzip.open(sys.argv[2], "wt", compresslevel=1) as out_file_h:
            for line in in_file_h:
                # Do processing on line here
                modified_line = line
                # End processing
                out_file_h.write(modified_line)

This is very slow, due to write being called for every line. This is the current implementation of write:

cpython/Lib/gzip.py

Line 272 in c379bc5

def write(self,data):

It:

Checks if the file is not closed
Checks if the correct mode is set
Checks if the file is not closed (again, but in a different way)
Checks if the data is bytes, bytearray or something that supports the buffer protocol
Gets the length
Compresses the data
updates the size and offset
updates the checksum

Doing this for every line written is very costly and creates a lot of overhead in Python calls. We spent a lot of time in Python and a lot less in the fast C zlib code that does the actual compression.

This problem is already solved on the read side. A _GzipReader object is used for reading. This is put in an io.BufferedReader which is used as the underlying buffer for GzipFile.read. This way, lines are read quite fast from a GzipFile without the checksum etc. being updated on every line read.

A similar solution should be written for write.
I volunteer (I have done some other work on gzip.py already), although I cannot give an ETA at this time.

gh-89550: Buffer GzipFile.write to reduce execution time by ~15% (#10…

pythongh-89550: Buffer GzipFile.write to reduce execution time by ~15% (

Merge branch 'main' into inlinecomp2

Merge branch 'main' into fixsendfamily

Merge branch 'main' into supermon

Merge branch 'main' into fixloadsuperspec

A PR has been merged, thanks for the report!

pythongh-129726: Break `gzip.GzipFile` reference loop

gh-129726: Break `gzip.GzipFile` reference loop (#130055)

pythongh-129726: Break `gzip.GzipFile` reference loop (pythonGH-130055)

[3.12] gh-129726: Break `gzip.GzipFile` reference loop (GH-130055) (#…

[3.13] gh-129726: Break `gzip.GzipFile` reference loop (GH-130055) (#…

Navigation Menu

Repository navigation

GzipFile.write should be buffered #89550

Linked PRs

9 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

GzipFile.write should be buffered #89550

Description

Linked PRs

Activity

rhpvorderman commented on Oct 6, 2021

9 remaining items

hugovk commented on Dec 11, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions