Repository navigation
Deprecate Py_HASH_EXTERNAL as it cannot be used for --with-hash-algorithm #141226
Description
Activity
- changed the title
[-]Py_HASH_EXTERNAL cannot be configure[/-][+]Py_HASH_EXTERNAL cannot be used[/+]on Nov 8, 2025 - changed the title
[-]Py_HASH_EXTERNAL cannot be used[/-][+]Py_HASH_EXTERNAL cannot be used for `--with-hash-algorithm`[/+]on Nov 8, 2025 I'm not convinced that this feature is needed/useful. Was it asked by users embedding Python?
It was part of PEP-456 (https://peps.python.org/pep-0456/#hash-function-selection), which I believe it takes precedence over whatever we discussed. I personally don't think it's useful but as it was implemented as part of the PEP, I don't think I can just leave it like this.
It happens that a PEP is not fully implemented, and that's ok. https://peps.python.org/pep-3118/ is an example of that:
"Not all features proposed here were implemented. Specifically:"
PyObject_CopyToObject was not added. [Additions to the struct string-syntax](https://peps.python.org/pep-3118/#struct-additions) were not added, except for ? (_Bool). PyObject_GetMemoryView is named [PyMemoryView_FromObject](https://docs.python.org/3/c-api/memoryview.html#c.PyMemoryView_FromObject).I suggest updating the PEP to document that external hashing was not implemented.
I suggest updating the PEP to document that external hashing was not implemented.
Unlike PEP-3118, the code path already and defining
Py_HASH_ALGORITHM Py_HASH_EXTERNALat compile time is possible for anyone building Python from source. It is just not easily exposed through the configure script (and it was really a matter of a single option). I cannot say it's not used but it's definitely possible to use it.One might want to provide an implementation of SipHash-1-3 using SIMD when possible or maybe there are private infrastructures that would do that when the build Python from scratch. I believe it's more work than just patching the sources directly but I cannot rule out this possibility.
Now, it would ease my life if we were to drop this constant but it's being declared in PyO3 (but doesn't seem to be used) so I don't know if it's possible to do anything in Rust as well. I can also say in the docs that it should not be used.
I suggest to deprecate this feature rather than fixing its implementation.
Ok, I will prepare a PR for the deprecation as well.
- changed the title
[-]Py_HASH_EXTERNAL cannot be used for `--with-hash-algorithm`[/-][+]Deprecate Py_HASH_EXTERNAL as it cannot be used for `--with-hash-algorithm`[/+]on Nov 8, 2025 AFAIK, the way to do this is configure with
CFLAGS='-DPy_HASH_ALGORITHM=Py_HASH_EXTERNAL', and then link with an implementation ofPyHash_Func.
For example you can make a very slow but relatively functional interpreter with:static Py_hash_t zerohash(const void *src, Py_ssize_t src_sz) { return 0; } PyHash_FuncDef PyHash_Func = {zerohash, "zerohash", 0, 0};
Looks like the only thing from the PEP that's not implemented is the configure integration?
Yes. That's what I said here as well: #141287 (comment) and implemented here https://github.com/picnixz/cpython/blob/58d86422acf9fd4d468344cb58ac19666df214fc/configure.ac#L3964-L3992
Removal has been scheduled for Python 3.19. If I hear complaints, I will revert the deprecation and ressurect the PR that fully implements the support via
configure.
Bug report
Bug description:
In PEP-456, we introduced the notion of interchangeable hash functions for strings, bytes and memoryviews. Historically, Python used FNV as its hash algorithm, switched to SipHash2-4 in 3.4 before making SipHash1-3 the default in 3.11. It is however possible for embedders to implement their own hashing algorithms but this requires Python to be configured with
--with-hash-algorithm=externaland linking Python with the appropriate additional library.Unfortunately, it's not possible to choose
externaland the only workaround would be to pass-DPy_HASH_ALGORITHM 0but this is neither documented nor tested (it may not even work if the macro is still defined afterwards).CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Linked PRs
Py_HASH_EXTERNALfor Unix platforms #141245