ZipFile.extractall fails if zipfile.Path object is created based on the ZipFile object

Bug report

The following code works fine with Python 3.8 and 3.9 but starts failing with Python 3.10.

It seems that zipfile.Path creates an entry in the zipfile.ZipFile object that does not exist in the underlying file and therefore makes zipfile.ZipFile.extractall fail.

import zipfile
import tempfile

path = tempfile.mktemp()

f = zipfile.ZipFile(path, "w")
f.writestr("data/test.txt", "test1")
f.writestr("data/test2.txt", "test2")
f.close()

dest = tempfile.mkdtemp()

p = zipfile.ZipFile(path)
src_zipfile_path = zipfile.Path(p, "data")
p.extractall(dest)

Your environment

Working version:

(python39) [moggi@mars cloudfluid]$ python --version
Python 3.9.16
(python39) [moggi@mars cloudfluid]$ python test.py
(python39) [moggi@mars cloudfluid]$

Failing versions:

3.10.9:

(python_310_2) [moggi@mars cloudfluid]$ python --version
Python 3.10.9
(python_310_2) [moggi@mars cloudfluid]$ python test.py
Traceback (most recent call last):
  File "/home/moggi/devel/cloudfluid/test.py", line 15, in <module>
    p.extractall(dest)
  File "/home/moggi/devel/envs/python_310_2/lib/python3.10/zipfile.py", line 1645, in extractall
    self._extract_member(zipinfo, path, pwd)
  File "/home/moggi/devel/envs/python_310_2/lib/python3.10/zipfile.py", line 1667, in _extract_member
    member = self.getinfo(member)
  File "/home/moggi/devel/envs/python_310_2/lib/python3.10/zipfile.py", line 1441, in getinfo
    raise KeyError(
KeyError: "There is no item named 'data/' in the archive"

and 3.11.0

(python311) [moggi@mars cloudfluid]$ python --version
Python 3.11.0
(python311) [moggi@mars cloudfluid]$ python test.py
Traceback (most recent call last):
  File "/home/moggi/devel/cloudfluid/test.py", line 15, in <module>
    p.extractall(dest)
  File "/home/moggi/devel/envs/python311/lib/python3.11/zipfile.py", line 1677, in extractall
    self._extract_member(zipinfo, path, pwd)
  File "/home/moggi/devel/envs/python311/lib/python3.11/zipfile.py", line 1699, in _extract_member
    member = self.getinfo(member)
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/moggi/devel/envs/python311/lib/python3.11/zipfile.py", line 1473, in getinfo
    raise KeyError(
KeyError: "There is no item named 'data/' in the archive"

If this is the expected new behavior I think the documentation for 3.10 should mention this breaking change and the zipfile.Path documentation might need an entry about this problem.

The zipfile.Path object is used in the original code to check if the "data/" directory exists in the zip file.

Linked PRs

@serhiy-storchaka

@serhiy-storchaka, @Yhg1s, @gpshead as zipfile experts.

@jaraco

@jaraco

This issue occurs because of the optimizations / tradeoffs made in attempting to extend an open file object. Here's what's going on:

zipfile.Path needs to be able to infer directories when there are none in the zipfile (i.e. the presence of data/test.txt implies the presence of data/ even though it's not explicitly a member of the zipfile).
To accomplish this goal, the CompleteDirs wraps the original zipfile.
Due to challenges with the lifecycle of a zipfile (which may have been created in memory or exist on disk as an open file), wrapping a Zipfile in a Path actually mutates its class, giving the original Zipfile object the CompleteDirs behavior.
In extractall, it's assumed that every item in the namelist is also a member, but with CompleteDirs, that's no longer true.

This report is the first I've seen that this (admittedly impure) behavior has caused any trouble.

I agree that at the very least, it's worthwhile documenting these limitations.

I did try to track down the origins of the change that led to this regression in behavior. I took a brief look at the readme for zipp, the forward/backport, and it doesn't mention any changes for Python 3.10 (now updated to reflect new findings).

Okay. I see the change occurred in ebbe803, bpo-40564/#84744. That bug pretty thoroughly describes the thought process and the tradeoffs considered.

I can think of a few ways to address the issue.

document it as unsupported behavior
document it as unsupported behavior, but provide a mechanism to restore the original object
add support for zipfile.Zipfile to be lenient to dirs not present
add support to CompleteDirs to return a virtual member info for an implied dir
CompleteDirs to override extractall to bypass the issue

I'm leaning toward (3) or (4). (4) has an advantage over all of the other options in that, if it can be implemented, it provides better compatibility for more use cases (not just ZipFile.extractall).

Add xfail test capturing missed expectation. Ref python/cpython#101566.

Override ZipFile.getinfo to supply a ZipInfo for implied dirs. Fixes p…

Update changelog. Ref python/cpython#101566.

In jaraco/zipp#90, I've drafted a patch implementing option 4, to be released as zipp 3.12.1. I'd like to get some feedback on the concept before porting that to cpython.

pythongh-101566: Sync with zipp 3.14.

gh-101566: Sync with zipp 3.14. (GH-102018)

[3.11] pythongh-101566: Sync with zipp 3.14. (pythonGH-102018).

[3.10] pythongh-101566: Sync with zipp 3.14. (pythonGH-102018).

[3.10] gh-101566: Sync with zipp 3.14. (GH-102018). (GH-102091)

[3.11] gh-101566: Sync with zipp 3.14. (GH-102018). (GH-102090)

Merge branch 'main' into inlinecomp2

Merge branch 'main' into inlinecomp

Merge branch 'main' into lockfreecp

This issue was closed back in Feb.

pythongh-101566: Sync with zipp 3.14. (pythonGH-102018)

Navigation Menu

Repository navigation

ZipFile.extractall fails if zipfile.Path object is created based on the ZipFile object #101566

Bug report

Your environment

Linked PRs

2 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ZipFile.extractall fails if zipfile.Path object is created based on the ZipFile object #101566

Description

Bug report

Your environment

Linked PRs

Activity

arhadthedev commented on Feb 4, 2023

gpshead commented on Feb 4, 2023

jaraco commented on Feb 5, 2023

jaraco commented on Feb 5, 2023

2 remaining items

jaraco commented on Jul 14, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions