Repository navigation
Add a file_digest() function in hashlib #89313
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2021-09-09.10:03:17.355>
labels = ['type-feature', 'library', '3.11']
title = 'Add a file_digest() function in hashlib'
updated_at = <Date 2022-03-22.15:41:04.324>
user = 'https://github.com/tarekziade'bugs.python.org fields:
activity = <Date 2022-03-22.15:41:04.324>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-09-09.10:03:17.355>
creator = 'tarek'
dependencies = []
files = []
hgrepos = []
issue_num = 45150
keywords = ['patch']
message_count = 14.0
messages = ['401457', '401461', '401462', '415138', '415139', '415141', '415320', '415321', '415324', '415326', '415336', '415356', '415754', '415793']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'christian.heimes', 'tarek', 'python-dev', 'Aur.Saraf', 'miss-islington']
pr_nums = ['28252', '31928', '31930', '32046']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45150'
versions = ['Python 3.11']
Activity
tarekziade commentedon Sep 9, 2021
I am proposing the addition of a very simple helper to return the hash of a file.
tiran commentedon Sep 9, 2021
Hey Tarek, long time no see!
the _sha256 module is optional, can be disabled and is not available in some distributions.
I also don't like to use sha256 has default. It's slow, even slower than sha512. Any default makes it also harder to upgrade to a better, more secure default in the future.
like hmac.new() a file_digest() should accept PEP-452-compatible arguments and hash name as digstmod argument, not just a callable.
a filename argument prevents users from passing in file-like objects like BytesIO.
4096 bytes chunk size is very conservative. The call overhead for read() and update() may dominate the performance of the function.
The hex argument feels weird.
In a perfect world, the hash and hmac objects should get an "update_file" method. The OpenSSL-based hashes could even release the GIL and utilize OpenSSL's BIO layer to avoid any Python overhead.
tarekziade commentedon Sep 9, 2021
Hey Christian, I hope things are well for you!
Thanks for all the precious feedback, I'll rework the patch accordingly
SonOfLilit commentedon Mar 14, 2022
Tarek,
Are you still working on this? Would you like me to take over?
Aur
tarekziade commentedon Mar 14, 2022
@aur, go for it, I started to implement it and got lost into the details for each backend..
SonOfLilit commentedon Mar 14, 2022
OK, I'll give it a go.
SonOfLilit commentedon Mar 16, 2022
PR contains a draft implementation, would appreciate some review before I implement the same interface on all builtin hashes as well as OpenSSL hashes.
5 remaining items
SonOfLilit commentedon Mar 16, 2022
I don't think HMAC of a file is a common enough use case to support, but I have absolutely no problem conceding this point, the cost of supporting it is very low.
I/O in C is a world of pain in general. In the specific case of
io.RawIOBaseobjects (non-buffered binary files) to my understanding it's not that terrible (am I right? Does my I/O code work as-is?). To my understanding, providing a fast path just for this case that calculates the hash without taking the GIL for every chunk would be very nice to have for many use cases.Now, we could just be happy with
file_digest()having anifforisinstance(io.RawIOBase)that chooses a fast code path silently. But since non-buffered binary files are so hard to tell apart from other types of file-like objects, as a user of this code I would like to have a way to say "I want the fast path, please raise if I accidentally passed the wrong things and got the regular path". We could havefile_digest('sha256', open(path, 'rb', buffered=0), ensure_fast_io=True), but I think for this use caseraw_file_digest('sha256', open(path, 'rb', buffered=0))is cleaner.In all other cases you just call
file_digest(), probably get the Python I/O and not the C I/O, and are still happy to have that loop written for you by someone who knows what they're doing.For the same reason I think the fast path should only support hash names and not constructors/functions/etc', which would complicate it because new-object-can-be-accessed-without-GIL wouldn't necessarily apply.
Does this make sense?
miss-islington commentedon Mar 22, 2022
New changeset 4f97d64 by Christian Heimes in branch 'main':
bpo-45150: Add hashlib.file_digest() for efficient file hashing (GH-31930)
4f97d64
tiran commentedon Mar 22, 2022
New changeset e03db6d by Christian Heimes in branch 'main':
bpo-45150: Fix testing under FIPS mode (GH-32046)
e03db6d
pablogsal commentedon Aug 13, 2022
@tiran Can you made a PR adding the
file_digestto the 3.11 what's new, please?pythongh-89313: Add hashlib.file_digest to whatsnew 3.11
gh-89313: Add hashlib.file_digest to whatsnew 3.11 (GH-95965)
pythongh-89313: Add hashlib.file_digest to whatsnew 3.11 (pythonGH-95965
gh-89313: Add hashlib.file_digest to whatsnew 3.11 (GH-95965)
calestyo commentedon Jun 23, 2023
Hey folks.
I know this is closed and perhaps I should simply file a new request... but would you consider to extend the interface of that function to (efficiently) calculate a file's hashsum for multiple algorithms (i.e. without reading it once for every algo)?
One could perhaps do so by accepting some array for
digestand return a dict where the alogo name is the key and the hashvalue the value.Or perhaps something smarter ^^
Cheers,
Chris.
gpshead commentedon Jun 23, 2023
Please file a new feature request issue here for that.