zipfile with multiprocessing: zipfile.BadZipFile #83544

Activity

1. maxime-lemonnier commented on Jan 16, 2020

   

   maxime-lemonniermannequin

   on Jan 16, 2020

   MannequinAuthor

   More actions

   zipfile sometimes throws zipfile.BadZipFile when opening the same zip file from multiple processes

   see attached file to reproduce the error. You'll need a zipfile with multiple files in it to reproduce.
2. maxime-lemonnier

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   on Jan 16, 2020
3. maxime-lemonnier commented on Jan 16, 2020

   

   maxime-lemonniermannequin

   on Jan 16, 2020

   MannequinAuthor

   More actions

   Here's my console output:

   python3 test_filesource.py
   lock
   file
   access_mode = file, nb processes = 1, res = 110289, 0.08039402961730957 ms/frame
   file
   access_mode = file, nb processes = 4, res = 110289, 0.32297492027282715 ms/frame
   lock
   access_mode = lock, nb processes = 4, res = 110289, 0.2950408458709717 ms/frame
   lock
   multiprocessing.pool.RemoteTraceback: 
   """
   Traceback (most recent call last):
     File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
       result = (True, func(*args, **kwds))
     File "/path/to/script/test_filesource.py", line 64, in read_small
       return fs_small[i%len(fs_small)][42]
     File "/path/to/script/test_filesource.py", line 55, in __getitem__
       data_bytes = self.access( lambda archive: archive.read(member))
     File "/path/to/script/test_filesource.py", line 27, in access_lock
       return f(self.archive)
     File "/path/to/script/test_filesource.py", line 55, in <lambda>
       data_bytes = self.access( lambda archive: archive.read(member))
     File "/usr/lib/python3.6/zipfile.py", line 1337, in read
       with self.open(name, "r", pwd) as fp:
     File "/usr/lib/python3.6/zipfile.py", line 1419, in open
       % (zinfo.orig_filename, fname))
   zipfile.BadZipFile: File name in directory '00000005.pkl' and header b'00000004.pkl' differ.
   """
   
   The above exception was the direct cause of the following exception:
   
   Traceback (most recent call last):
     File "/path/to/script/test_filesource.py", line 90, in <module>
       f(4, "lock") #crash
     File "/path/to/script/test_filesource.py", line 81, in f
       for i in pool.imap_unordered(read_small, frames):
     File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next
       raise value
   zipfile.BadZipFile: File name in directory '00000005.pkl' and header b'00000004.pkl' differ.
4. ezio-melotti

   transferred this issue fromon Apr 10, 2022
5. serhiy-storchaka

   added this to Zipfile issues 📦and Multiprocessing issueson May 19, 2022
6. vstinner

   added

   topic-multiprocessing

   on Jun 6, 2022
7. AlexWaygood

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   and removed

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   on Jul 10, 2022
8. gpshead commented on Dec 21, 2023

   

   gpshead

   on Dec 21, 2023

   Member

   More actions

   It appears you are attempting to use the same already opened zipfile instance from multiple different processes at once. This won't even work with multiprocessing start methods other than "fork" as the other processes would not inherit the open file descriptors and the zipfile instance would fail to pickle/unpickle to go to the other process.

   In the "fork"ing case, an existing zipfile instance already has an open file and is thus not safe for use from those processes as they'll all be attempting to use the same underlying inherited open file descriptor at once.

   zipfile is not going to try to prevent this.

   Recommendation: Open a new zipfile instance from within each of the processes. parallel reading will work fine then as they no longer share any state.

   Other recommendation: Do not use the "fork" multiprocessing start method. use "forkserver" or "spawn". you'll have far fewer problems. (the default start method on posix systems will change to one of those in the future)
9. gpshead

   closed this as not plannedon Dec 21, 2023
10. github-project-automation

    moved this to Done in Zipfile issues 📦on Dec 21, 2023
11. github-project-automation

    moved this to Done in Multiprocessing issueson Dec 21, 2023