support ZIP files with zeroed out fields (e.g. for reproducible builds)

@tiran

BPO	43547
Nosy	@tiran, @jondo, @obfusk, @eighthave
PRs	bpo-43547: add zeroed=False option to ZipFile #24979

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-03-18.21:25:38.236>
labels = ['type-feature', 'library', '3.10']
title = 'support ZIP files with zeroed out fields (e.g. for reproducible builds)'
updated_at = <Date 2021-06-22.13:49:25.963>
user = 'https://github.com/eighthave'

bugs.python.org fields:

activity = <Date 2021-06-22.13:49:25.963>
actor = 'obfusk'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-03-18.21:25:38.236>
creator = 'eighthave'
dependencies = []
files = []
hgrepos = []
issue_num = 43547
keywords = ['patch']
message_count = 11.0
messages = ['389040', '389041', '389338', '389339', '389343', '389348', '389349', '389382', '389392', '389441', '396332']
nosy_count = 4.0
nosy_names = ['christian.heimes', 'jondo', 'obfusk', 'eighthave']
pr_nums = ['24979']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue43547'
versions = ['Python 3.10']

It is now standard for Java JARs and Android APKs (both ZIP files) to zero out lots of the fields in the ZIP header. For example:

each file entry has the date set to zero
the create_system is always set to zero on all platforms

zipfile currently cannot create such ZIPs because of two small restrictions that it introduced:

must use a tuple of 6 values to set the date
forced create_system value based on sys.platform == 'win32'
maybe other fields?

I lump these together because it might make sense to handle this with a single argument, something like zero_header=True. The use case is for working with ZIP, JAR, APK, AAR files for reproducible builds. The whole build system for F-Droid is built in Python. We need to be able to copy the JAR/APK signatures in order to reproduce signed builds using only the source code and the signature files themselves. Right now, that's not possible because building a ZIP with Python's zipfile cannot zero out the ZIP header like other tools can, including Java.

I just found another specific example in _open_to_write(). 0 is a valid value for zinfo.external_attr. But this code always forces 0 to something else:

        if not zinfo.external_attr:
            zinfo.external_attr = 0o600 << 16  # permissions: ?rw
```-------

I've created a draft PR; RFC :)

Also:

setting the date to (1980,0,0,0,0,0) already works;
the main issue seems to be that external_attr cannot be 0 atm.

external_attr == 0 may cause issues with permissions.

That may be true in some scenarios, but not being able to set it to 0 means you can't create identical files to those produced by other tools -- like those used to generate APKs -- which do in fact set it to 0.

The __getattr__ hack is not needed. You can reset the flags in a different, more straight forward way:

class ReproducibleZipInfo(ZipInfo):
    __slots__ = ()

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._reset_flags()

    @classmethod
    def from_file(cls, *args, **kwargs):
        zinfo = super().from_file(*args, **kwargs)
        zinfo._reset_flags()
        return zinfo

    def _reset_flags(self):
        self.date_time = (1980, 0, 0, 0, 0, 0)
        self.create_system = 0
        self.external_attr = 0

>>> zinfo = ReproducibleZipInfo.from_file("/etc/os-release")
>>> zinfo.external_attr
0
>>> zinfo.create_system
0
>>> zinfo.date_time
(1980, 0, 0, 0, 0, 0)

I think it makes also sense to replace hard-coded ZipInfo class with dispatcher attribute on the class:

@@ -1203,6 +1211,7 @@ class ZipFile:
 
     fp = None                   # Set here since __del__ checks it
     _windows_illegal_name_trans_table = None
+    zipinfo_class = ZipInfo
 
     def __init__(self, file, mode="r", compression=ZIP_STORED, allowZip64=True,
                  compresslevel=None, *, strict_timestamps=True):
@@ -1362,7 +1371,7 @@ def _RealGetContents(self):
                 # Historical ZIP filename encoding
                 filename = filename.decode('cp437')
             # Create ZipInfo instance to store file information
-            x = ZipInfo(filename)
+            x = self.zipinfo_class(filename)

The __getattr__ hack is not needed. You can reset the flags in a different, more straight forward way

As mentioned, ZipFile._open_to_write() will modify the ZipInfo's .external_attr when it is set to 0.

I just found another specific example in _open_to_write(). 0 is a valid value for zinfo.external_attr. But this code always forces 0 to something else:
if not zinfo.external_attr:
    zinfo.external_attr = 0o600 \<\< 16  # permissions: ?rw\-------

Your alternative doesn't seem to take that subsequent modification into account.

For full reproducible builds you may have to write files to zipfiles in a well-defined order.

That already works fine now, we've been doing that with Python for years. But that leaves it up to the implemented to do. I suppose zipfile could provide a method to sort entries, but that's out of scope for this issue IMHO.

https://github.com/obfusk/apksigcopier currently produces reproducible ZIP files identical to those produced by apksigner using this code:

DATETIMEZERO = (1980, 0, 0, 0, 0, 0)


class ReproducibleZipInfo(zipfile.ZipInfo):
    """Reproducible ZipInfo hack."""

    _override = {}  # type: Dict[str, Any]

    def __init__(self, zinfo, **override):
        if override:
            self._override = {**self._override, **override}
        for k in self.__slots__:
            if hasattr(zinfo, k):
                setattr(self, k, getattr(zinfo, k))

    def __getattribute__(self, name):
        if name != "_override":
            try:
                return self._override[name]
            except KeyError:
                pass
        return object.__getattribute__(self, name)


class APKZipInfo(ReproducibleZipInfo):
    """Reproducible ZipInfo for APK files."""

    _override = dict(
        compress_type=8,
        create_system=0,
        create_version=20,
        date_time=DATETIMEZERO,
        external_attr=0,
        extract_version=20,
        flag_bits=0x800,
    )


def patch_meta(...):
    ...
    with zipfile.ZipFile(output_apk, "a") as zf_out:
        info_data = [(APKZipInfo(info, date_time=date_time), data)
                     for info, data in extracted_meta]
        _write_to_zip(info_data, zf_out)


if sys.version_info >= (3, 7):
    def _write_to_zip(info_data, zf_out):
        for info, data in info_data:
            zf_out.writestr(info, data, compresslevel=9)
else:
    def _write_to_zip(info_data, zf_out):
        old = zipfile._get_compressor
        zipfile._get_compressor = lambda _: zlib.compressobj(9, 8, -15)
        try:
            for info, data in info_data:
                zf_out.writestr(info, data)
        finally:
            zipfile._get_compressor = old

@jayqi

TIL https://pypi.org/project/repro-zipfile/ (and https://pypi.org/project/repro-tarfile/) by @jayqi, which led me to wonder whether this functionality belongs in stdlib ZipFile, which in turn led me here. Looks like this issue has stalled, so posting this comment to

show that there is still evidence of demand for this feature
ask the stdlib maintainers if there is still interest in this, and if the PyPI package (which was not yet discussed here, appears to be well-written and proven in production) provides more helpful prior art
ask if there's anything I can do to help :)

Navigation Menu

Repository navigation

support ZIP files with zeroed out fields (e.g. for reproducible builds) #87713

31 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

support ZIP files with zeroed out fields (e.g. for reproducible builds) #87713

Description

Activity

eighthave commented on Mar 18, 2021

eighthave commented on Mar 18, 2021

obfusk commented on Mar 22, 2021

31 remaining items

obfusk commented on Mar 22, 2021

tiran commented on Mar 23, 2021

obfusk commented on Mar 23, 2021

eighthave commented on Mar 24, 2021

obfusk commented on Jun 22, 2021

ctcjab commented on Sep 18, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions