Stream callbacks retake the GIL on every read/seek/write: multithreaded signing is slower than single-threaded #348

Summary

Every read, seek and write the native library does on a Stream goes through a Python ctypes callback, and each callback retakes the GIL. An asset needs tens of thousands of them: one or more per PNG chunk or BMFF box. As a result:

• Signing from several threads is slower than from one. Signatures queue on the GIL inside their callbacks instead of running in parallel.
• Any busy Python thread stalls a signature for tens of seconds. Each callback waits for the busy thread to release the GIL (up to the 5 ms switch interval), so a signature that takes 0.1 s alone takes 30 to 100 s next to a CPU-bound thread.

This affects any multithreaded use: thread pools, web servers, asyncio.to_thread, or worker processes that run other Python threads.

Measurements

Builder.sign(signer, mime, src, dst) with src and dst regular files opened by Python, an ES256 signer without TSA, and one input copy per thread. Run on main (ae8628a, c2pa-rs v0.91.5), macOS arm64, 14 cores, Python 3.12. Throughput in signatures per second:

Callback counts. For one signature of the 4 KB-chunk PNG, Builder.sign makes about 37,600 readinto calls (average 0.4 KiB), 35,700 seeks and 2,000 writes. A Reader validating the result makes about 24,000 reads and 24,000 seeks more. The same image re-encoded with 64 KB chunks needs about 14 times fewer. The 72.7 MB MP4 needs about 20,000 callbacks in all.

Processes scale where threads don't. The same workload in a process pool (4 KB-chunk PNG) gives 9.8/s with 1 process, 16.9/s with 2 and 24.1/s with 4. So the native side does not serialize: the only shared state on the Rust side is the handle-registry mutex, held briefly. The bottleneck is the Python callbacks.

We saw the same pattern on Linux x86-64 containers: with 8 concurrent signatures per 1-CPU process, the sign step of a 4K PNG went from about 1 s to 16–24 s.

Proposed fix

When the object wrapped by Stream is a regular file on a 64-bit POSIX platform, give c2pa_create_stream libc read / lseek / write as callbacks, with a private dup() of the file descriptor as the context. The native call then never re-enters Python, and the GIL stays released for its whole duration.

• The duplicate starts at the Python object's position (after a flush()), and the final position is given back on close().
• Closing the Python file while the Stream is alive cannot make the native side read another file that reused the descriptor number.
• In-memory streams, pipes and sockets, append-mode files, and other platforms keep the Python callbacks.
• Stream.native_file_io = False opts out.

The same benchmark with this change:

I'll open a PR with this change and its tests. The full unit and threaded suites pass with it.

A more general fix would belong in c2pa_c_ffi: a stream constructor that takes a path or a file descriptor, or buffered I/O around the callbacks. Every binding would then benefit. I'm happy to look into that too if it's the preferred direction.