Repository navigation
Use zlib-ng (fast!) rather than mainline stale zlib in binary releases #91349
Description
| BPO | 47193 |
|---|---|
| Nosy | @gpshead, @pfmoore, @tjguk, @zware, @zooba, @corona10, @arhadthedev |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2022-04-01.19:19:24.173>
labels = ['3.11', 'OS-windows', 'performance']
title = 'Use zlib-ng rather than zlib in binary releases'
updated_at = <Date 2022-04-02.09:48:33.061>
user = 'https://github.com/gpshead'bugs.python.org fields:
activity = <Date 2022-04-02.09:48:33.061>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2022-04-01.19:19:24.173>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 47193
keywords = []
message_count = 1.0
messages = ['416508']
nosy_count = 7.0
nosy_names = ['gregory.p.smith', 'paul.moore', 'tim.golden', 'zach.ware', 'steve.dower', 'corona10', 'arhadthedev']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue47193'
versions = ['Python 3.11']
Activity
gpshead commentedon Apr 1, 2022
zlib-ng is an optimized zlib library with better performance on most architectures (with contributions from the likes of Google, Cloudflare, and Intel). It is API compatible with zlib. https://github.com/zlib-ng/zlib-ng
I believe the only platform we don't use the OS's own zlib on is Windows so I'm tagging this issue Windows.
gpshead commentedon Jun 17, 2022
if this hasn't happened in the 3.11 betas this is presumably bumped to 3.12.
zooba commentedon Jul 17, 2022
I started taking a look at this and it seems like we can build it without having to worry about their build system by renaming the
zconf.h.inandzconf-ng.h.infiles to remove the.in, and settingZLIB_COMPATpreprocessor in pythoncore.vcxproj (as well as referencing the files).Running test_zlib the only failures seem to be testing for certain failures that no longer fail, but I've got no idea how important they are. Also no indication of the performance impact, or anything else that may change (e.g. new DLL exports, etc.), but it certainly does seem like a feasible drop-in replacement.
rhpvorderman commentedon Aug 1, 2022
@zooba which failures are these? I have accumulated quite some experience with the zlib/gzip formats due to working on python-isal, bindings for the ISA-L library that speeds up zlib-compatible compression by rewriting the algorithms in x86 Assembly language. It is quite good, but not suitable for a drop-in replacement, hence the python-isal project.
Having said that I'd love to help out with anything zlib related in CPython.
zooba commentedon Aug 1, 2022
I knew I should've kept better track of the changed error messages 😄
Skimming through the tests, I'm pretty sure
test_wbitsfailed in a few places because we expected some "invalid window size" errors that aren't errors with zlib-ng. I don't think any were critical, but I don't know the usage of the library well enough to be sure (e.g. do people expect these errors and change behaviour? or are they always just developer error and cause changes to just make things work?)64 remaining items
gpshead commentedon Mar 27, 2025
Contributions to the linux build as in configure.ac & Makefile.pre.in are welcome so that when a recent enough version of zlib-ng is found, it is preferred over vanilla zlib.
pythongh-91349: Expose the crc32 function from the lzma library
morotti commentedon Apr 17, 2025
quick update,
I have attempted to adjust the linux build to detect and use zlib-ng, to no success.
the .configure script eludes me.
there might only be 10 people in the world who understand any of it.
any help would be greatly appreciated.
liblzma just had a new release a few days ago (xz-utils package on linux). https://github.com/tukaani-project/xz/releases
the latest liblzma release 5.8.x is using hardware CRC instructions, it is now as fast as zlib-ng and libdeflate.
I have submitted a PR to expose crc32 from lzma.
after it is merged, I shall do more PR to pick the optimal crc function between zlib/zlibng/liblzma depending on what is available.
nmoinvaz commentedon Apr 17, 2025
It would be great to see what the stats are compared to zlib-ng. It has optimizations for different architectures and even instruction sets. So it will perform well on older hardware if that is necessary for your use case.
pythongh-91349: Adjust default compression level to 6 (down from 9) i…
pythongh-91349: Adjust default compression level to 6 (down from 9) i…
pythongh-91349: Expose the crc32 function from the lzma library
gh-91349: Adjust default compression level to 6 (down from 9) in gzip…
pythongh-91349: Adjust default compression level to 6 (down from 9) i…
pythongh-91349: Adjust default compression level to 6 (down from 9) i…
pythongh-91349: Expose the crc32 function from the lzma library
morotti commentedon May 21, 2026
Just a quick update since the Python interpreter transitioned to zlib-ng on Windows, I've performed extensive benchmarks and analysis since we did the transition, you can read here, it's a massive improvement! ->
Faster Python 3.14: faster compression and fixing 28-year-old bugs
rhpvorderman commentedon Jul 21, 2026
Thanks @morotti for the write up. Very interesting. It also seems that my side project of python-zlib-ng is now redundant for windows 3.14 and higher. Less maintenance for me in the future. I hope that zlib-ng gets shipped by distro's soon.