Sparse file in archive triggers "OSError: [Errno 28] No space left on device: '...' " during scan #32

Activity

1. goekDil

   changed the title [-]"OSError: [Errno 28]" No space left on device: '...' during scan[/-] [+]"OSError: [Errno 28] No space left on device: '...' " during scan[/+] on Sep 29, 2021
2. pombredanne commented on Sep 29, 2021

   

   pombredanne

   on Sep 29, 2021

   Member

   More actions

   Would this be with https://github.com/moby/moby/archive/refs/tags/v20.10.5.tar.gz ?
3. pombredanne commented on Sep 30, 2021

   

   pombredanne

   on Sep 30, 2021

   Member

   More actions

   And I had not seen your attachment! so I am good wih the code link you provided.

   So I think this is sparse tarball issue which is 60GB extracted and only 5KB otherwise, unextracted, and not even compressed

   Can you test scanning that one single 5KB file?
   https://github.com/moby/moby/blob/363e9a88a11be517d9e8c65c998ff56f774eb4dc/vendor/archive/tar/testdata/gnu-sparse-big.tar

   And can you detail what pre-processing you apply on this. Did you call extractcode first?
   Is there a place where I can see your code? I am always interested in how ScanCode is integrated!
4. pombredanne commented on Sep 30, 2021

   

   pombredanne

   on Sep 30, 2021

   Member

   More actions

   Note that this could be related to aboutcode-org/scancode-toolkit#2431 where @Angi2412 and @avishmehta68710 both mentioned having a similar issue or at least this comment aboutcode-org/scancode-toolkit#2431 (comment) references the same code:

   I could observe the same behaviour and error message with this file: moby v20.10.5.
5. goekDil commented on Sep 30, 2021

   

   goekDil

   on Sep 30, 2021

   Author

   More actions

   Yes, I apply extractcode first! Unfortunately, there is no place where you can see my code, but I can provide a minimalexample.py:

     from scancode import cli
     import multiprocessing
     import logging
     from extractcode import extract
     
     def main():
     
         extract_log = extract.extract(location='gnu-sparse-big.tar', recurse=True)
         i = 0
         for e in extract_log:
             print(e)
     
         rc, results = cli.run_scan(
             'gnu-sparse-big.tar-extract', license=True, copyright=True,
             return_results=True, processes=14, verbose=True, quiet=False,
             timeout=46800)
     
         print(rc, results)
     
     if __name__ == "__main__":
         main()
   

   As you proposed, I tested the one single 5KB file and the exact same Error occurs:

   
   (docker logs) (Linux and Windows with docker)

   Depending on the machine I work on

   • I either experience the same Issue as in Timeout error despite high timeout scancode-toolkit#2431 (Linux with docker or run minimalexample.py as standalone python script)
   • or there appears the Issue "OSError: [Errno 28] No space left on device" as I mentioned above (Windows with docker, as well as Linux with docker)
6. pombredanne commented on Oct 8, 2021

   

   pombredanne

   on Oct 8, 2021

   Member

   More actions

   So this is a sparse file issue. The short term workaround may be to ignore the gnu-sparse-big.tar file entirely.
7. goekDil commented on Oct 8, 2021

   

   goekDil

   on Oct 8, 2021

   Author

   More actions

   Thank you very much! After further analysis, I also came to the conclusion that this is a sparse file issue.
8. goekDil

   closed this as completedon Oct 8, 2021
9. pombredanne commented on Oct 8, 2021

   

   pombredanne

   on Oct 8, 2021

   Member

   More actions

   I still may want to keep this open for now and transfer the issue to extractcode... as we may want to have a special processing for sparse files
10. pombredanne

    reopened this on Oct 8, 2021
11. pombredanne

    changed the title [-]"OSError: [Errno 28] No space left on device: '...' " during scan[/-] [+]Sparse file in archive triggers "OSError: [Errno 28] No space left on device: '...' " during scan[/+] on Oct 8, 2021
12. pombredanne

    transferred this issue fromaboutcode-org/scancode-toolkiton Oct 8, 2021
13. pombredanne commented on Oct 8, 2021

    

    pombredanne

    on Oct 8, 2021

    Member

    More actions

    Done... now in extractcode!
14. pombredanne

    added a commit that references this issue on Oct 8, 2021

    Merge pull request #32 from nexB/install-from-thirdparty-dir

    ...

    fa13562