Stream bindings are slow and broken #240

Activity

1. jonathanunderwood

   added

   help wanted

   streamRelated to stream compression bindingsRelated to stream compression bindings

   on Feb 17, 2022
2. utam-1 commented on Oct 4, 2026

   

   utam-1

   on Oct 4, 2026

   More actions

   @jonathanunderwood, If this issue is still open, can I work on this? I went through _stream.c, and based on the suggested approach you provided above and my understanding of the code, I observed that it is doing most of the heavy lifting work like memory management, buffer tracking etc.
   Instead of this we could shift the risky memory and buffer management logic to Python code and use the C code for calling the relevant compression algos.
   This would be the first task that needs to be done. ( As per my understanding of the issue and overall codebase ).

   If you want, I can open up a PR and start working on it.
3. jonathanunderwood commented on Oct 4, 2026

   

   jonathanunderwood

   on Oct 4, 2026

   MemberAuthor

   More actions

   Yes, all help welcome! I have very limited time to work on this package currently.

   I think the existing bindings are too opinionated, focusing on the double buffer and ring buffer implementations. I think a better approach would be to have slim C bindings that expose the relevant primitives such that ring buffer, double buffer, or any other approach to leveraging them is done at the python layer. I think you're thinking along similar lines.

   I think it would be helpful to first sketch out an approach for what the C extension API would look like as a starting point.
4. utam-1 commented on Oct 5, 2026

   

   utam-1

   on Oct 5, 2026

   More actions

   Yes, all help welcome! I have very limited time to work on this package currently.

   I think the existing bindings are too opinionated, focusing on the double buffer and ring buffer implementations. I think a better approach would be to have slim C bindings that expose the relevant primitives such that ring buffer, double buffer, or any other approach to leveraging them is done at the python layer. I think you're thinking along similar lines.

   I think it would be helpful to first sketch out an approach for what the C extension API would look like as a starting point.

   Thanks, so I was thinking something in the following line, expose all the streaming primitives of LZ4, leaving the stuff related to memory management to Python. C extension API would roughly be something like this :

   // LZ4 requires memory to handle its mathematical state, we'll create the same with C extension and hand it over to our Python code which would decide when to get rid of it.
   FUNCTION create_state() -> returns a Ref_ptr
   FUNCTION free_state(Ref_ptr)
   
   // Tells the compressor what data came previously, so it can find matches.
   FUNCTION set_dictionary(Token, Bytes previous_data)
   
   // Python code would identify the memory to be processed and hand it over to C code for processing, compress and decompress won't do any buffering in this case.
   FUNCTION compress(Token, Bytes data_to_compress) -> returns Bytes compressed_data
   FUNCTION decompress(Token, Bytes data_to_decompress) -> returns Bytes uncompressed_data
   
   

   This is a rough sketch of what I've thought about, I'll open up a PR too soon should you wish to proceed with this, or if you want to add anything else.