Skip to content

tls: pass raw digest bytes to prehashed sign() (fix fastecdsa 3.x) - #3

Open
ikegam wants to merge 1 commit into
3v1n0:masterfrom
ikegam:fix-fastecdsa-prehashed-bytes
Open

ikegam wants to merge 1 commit into
3v1n0:masterfrom
ikegam:fix-fastecdsa-prehashed-bytes

Conversation

@ikegam

@ikegam ikegam commented Jul 15, 2026

Copy link
Copy Markdown

Problem

On modern fastecdsa (3.x), sensor pairing/initialization fails with:

TypeError: Prehashed message must be bytes, got <class 'str'>

fastecdsa.ecdsa.sign(..., prehashed=True) now requires the prehashed
message to be bytes and hexlifies it internally (see _hex_digest in
fastecdsa's ecdsa.py). However Tls.make_cert_verify() passes an
already hex-encoded str (hexlify(buf).decode()), which trips the new
type check and aborts the TLS handshake during validity-sensors-tools
initialization / pairing.

Fix

Pass the raw digest bytes (buf) directly. fastecdsa hexlifies it
internally, producing the identical hex digest that was previously
passed by hand, so the resulting signature is unchanged — this is
behavior-preserving on old fastecdsa and fixes it on new fastecdsa.

Testing

Verified end-to-end on a Validity 138a:0090 (VFS7500) sensor: with this
change, validity-sensors-tools -t initializer completes factory reset,
firmware upload, pairing and calibration successfully, and fprintd
enroll/verify then work via the libfprint vfs0090 TOD driver. Before the
change, initialization died at make_cert_verify with the TypeError
above.

fastecdsa's sign(..., prehashed=True) now requires the prehashed
message to be bytes and hexlifies it internally (see fastecdsa
ecdsa._hex_digest), raising:

    TypeError: Prehashed message must be bytes, got <class 'str'>

make_cert_verify() was passing an already hex-encoded str
(hexlify(buf).decode()). Pass the raw digest bytes (buf) instead;
fastecdsa produces the identical hex digest internally, so the
signature is unchanged. This breaks sensor pairing/initialization
on modern fastecdsa (3.x).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant