Repository navigation
Renaming of tmp folder to data fails due to antivirus scans on Windows #191
Description
We have been bagging on network shares lately using bagit-python as opposed to working on local forensic workstations. As a result, some new issues have arisen, largely with bagit failing due to not having permission to rename the tmp folder to 'data'. We believe that this is due to antivirus scans (Windows Defender) on the network placing a lock on the rename process for a few seconds.
PermissionError: [WinError 5] Access is denied: 'X:\\storage\\workingFiles\\bag_tests\\pythonforktest\\000929796\\tmpe1ehkzxc' -> 'data'
Looking at old issues, I can see that this error has popped up here before: #134 so it might be a bit more widespread than I would have initially thought.
I have added some try/except code to a local fork of bagit-python and it has been handling these issues really well. Anecdotally, the issue arises for us about 20% of the time in a test of about 400 bags totalling 100gigs, and usually a retry after 10 seconds fixes the issue, with a rare few instances needed two retries. In the issue linked to above, this 'retry' solution appears to have worked as well as they were able to run bagit successfully again after the initial fail.
I will follow up with the pull request.
Best,
Kieran O'Leary
Digital Repository Services Manager
National Library of Ireland
Activity
acdha commentedon Aug 14, 2025
Thanks, that's an interesting challenge since it's technically an environmental problem but a very real one anyone on Windows could encounter. Your code is very simple so it's probably better than adding a dependency on stamina but the Python packaging situation has improved massively since @edsu was originally working on bagit.py and I wonder whether we might want to start using retries with backoff, etc. to handle other errors as well. I'm thinking that the answer might be to merge #192 now but consider how often it is the case that a transient permission or I/O error could be retried within a modest period of time as I know people have had issues with things like network filesystems in the past, too.
kieranjol commentedon Aug 14, 2025