Skip to content

Rewrite & cleanup HACL*-based extension modules #135532

Closed
Closed
Rewrite & cleanup HACL*-based extension modules#135532
Assignees
picnixz
Labels
extension-modulesC modules in the Modules dirtype-refactorCode refactoring (with no changes in behavior)

Description

@picnixz
picnixz
opened on Jun 15, 2025

I recently rewrote the _blake2 module. I want to the same for other modules where I can align the naming of functions across the different modules. The reason why I'm doing this is because it's becoming harder to make "similar" changes everywhere. In general, if I need to change something in MD5, then I also need to change it in SHA1/SHA2/SHA3, and it's easier if the code looks similar elsewhere.

There are some places that need to be updated because they are dead code, e.g.:

/*[clinic input]
module _sha2
class SHA256Type "SHA256object *" "&PyType_Type"
class SHA512Type "SHA512object *" "&PyType_Type"
[clinic start generated code]*/

should be

/*[clinic input]
module _sha2
class SHA256Type "SHA256object *" "clinic_state()->sha256_type"
class SHA512Type "SHA512object *" "clinic_state()->sha512_type"
[clinic start generated code]*/

Linked PRs

Bug fixes (3.14+)

Backported PRs (HMAC-only)

Abandoned

Activity

  1. picnixz
    self-assigned this
    on Jun 15, 2025
  2. picnixz
    added
    extension-modulesC modules in the Modules dir
    type-refactorCode refactoring (with no changes in behavior)
    on Jun 15, 2025

  3. picnixz commented on Jun 19, 2025

    @picnixz
    picnixz
    on Jun 19, 2025
    MemberAuthor

    I've rewritten SHA-2 and SHA-3 modules with generic macros but I'm torn between the now cleaner modules and the fact that I entirely rewrote them. SHA-2 is smaller but SHA-3 is larger.

  4. picnixz commented on Jun 19, 2025

    @picnixz
    picnixz
    on Jun 19, 2025
    MemberAuthor

    Offline I found other stuff that need to be updated independently:

    • SHA-3 types in Argument Clinic code.
    • MD-5 digests computations don't use _Py_strhex.
    • Useless memory allocation for 0-length SHAKE hashes (we can switch conditions checks)
    • Usage of unsigned char where we could use uint8_t: HACL* code expects an uint8_t so we can consistently use it.
    • Inconsistentcies in naming classes: we have _sha2.SHA256Type vs _sha3.sha3_224. Note SHA-2 doesn't expose a __new__ method but SHA-3 does. MD-5/SHA-1/SHA-1 don't have a tp_new slot but SHA-3 and BLAKE-2 do. I don't know which one is the best, but I don't think we should have both.
  5. picnixz
    added a commit that references this issue on Jun 20, 2025

    gh-135532: simplify handling of HACL* errors in `_hmac` (#135740)

    2dbada1
  6. miss-islington
    added a commit that references this issue on Jun 20, 2025

    pythongh-135532: simplify handling of HACL* errors in `_hmac` (python…

    fdad74d
  7. picnixz
    added 2 commits that reference this issue on Jun 20, 2025

    [3.14] gh-135532: simplify handling of HACL* errors in `_hmac` (GH-13…

    d057cb4

    gh-135532: update `Modules/_hacl/python_hacl_namespaces.h` (#135741)

    57dba7c
  8. miss-islington
    added a commit that references this issue on Jun 20, 2025

    pythongh-135532: update `Modules/_hacl/python_hacl_namespaces.h` (pyt…

    adeb79f
  9. picnixz
    added 5 commits that reference this issue on Jun 20, 2025

    gh-135532: use `_Py_strhex` in HACL-MD5's `hexdigest` (#135742)

    eec7a8f

    [3.14] gh-135532: update `Modules/_hacl/python_hacl_namespaces.h` (GH…

    28fc24e

    gh-135532: optimize calls to `PyMem_Malloc` in SHAKE digest computati…

    d08b4b2

    gh-135532: cleanup clinic `module` directives for cryptographic modul…

    621a8bd

    gh-135532: use `defining_class` for copying BLAKE-2 and SHA-3 objects (…

    ef4fc86

  10. 7 remaining items

  11. Pranjal095
    added a commit that references this issue on Jul 12, 2025

    pythongh-135532: use `defining_class` for copying BLAKE-2 and SHA-3 o…

    9b530a9

  12. picnixz commented on Jul 13, 2025

    @picnixz
    picnixz
    on Jul 13, 2025
    MemberAuthor

    I think I've done enough cleanup. The PR that is currently a draft for the module's state is stale and I won't merge it because it's not that important IMO. Other modules also suffer from similar constructions, so the best plan is to add some internal helpers for correctly fetching the module's def/state from a final heap type.

  13. picnixz
    closed this as completedon Jul 13, 2025
  14. taegyunkim
    added 6 commits that reference this issue on Aug 4, 2025

    pythongh-135532: simplify handling of HACL* errors in `_hmac` (python…

    0a9b1bb

    pythongh-135532: update `Modules/_hacl/python_hacl_namespaces.h` (pyt…

    d962c88

    pythongh-135532: use `_Py_strhex` in HACL-MD5's `hexdigest` (python#1…

    f262490

    pythongh-135532: optimize calls to `PyMem_Malloc` in SHAKE digest com…

    c5183d7

    pythongh-135532: cleanup clinic `module` directives for cryptographic…

    6a4d724

    pythongh-135532: use `defining_class` for copying BLAKE-2 and SHA-3 o…

    6c18125
  15. Agent-Hellboy
    added 6 commits that reference this issue on Aug 19, 2025

    pythongh-135532: simplify handling of HACL* errors in `_hmac` (python…

    100306b

    pythongh-135532: update `Modules/_hacl/python_hacl_namespaces.h` (pyt…

    2b72f68

    pythongh-135532: use `_Py_strhex` in HACL-MD5's `hexdigest` (python#1…

    e682bcb

    pythongh-135532: optimize calls to `PyMem_Malloc` in SHAKE digest com…

    b6f87f1

    pythongh-135532: cleanup clinic `module` directives for cryptographic…

    e356a1c

    pythongh-135532: use `defining_class` for copying BLAKE-2 and SHA-3 o…

    4dc62a2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

extension-modulesC modules in the Modules dirtype-refactorCode refactoring (with no changes in behavior)

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions