You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Missing support for dictionary compression (unstable, non-public API) #256
I'm looking into implementing this but hit an upstream roadblock: lz4/lz4#791. The frame dictionary APIs are still considered unstable (despite their age) and are only available with statically linked copies of lz4. Today, python-lz4 supports both but prefers the dynamically-linked system copy of lz4 if available.
We can fail at runtime still if we are using a dynamically linked copy, but how lz4 was built into the package is pretty opaque to the user.
@jonathanunderwood-- do you think having this as experimental only and forcing static links for experimental builds is a good middle ground? This seems like it only isn't stable because there hasn't been a major lz4 release in the last three years.
@jonathanunderwood-- do you think having this as experimental only and forcing static links for experimental builds is a good middle ground? This seems like it only isn't stable because there hasn't been a major lz4 release in the last three years.
I'd consider it if someone submitted a PR and the added complexity in setup.py etc wasn't high and the submitter was prepared to stick around and support it. I have very limited bandwidth to support this package, and this really isn't a high priority compared to the other stuff that is outstanding (fixing/rewriting the frame bindings, Aarch64 builds, type hinting). But honestly, before that, I'd need to see convincing numbers demonstrating it's actually worth it.
changed the title [-]Don't support dictionary compression[/-][+]Missing support for dictionary compression (unstable, non-public API)[/+]on Dec 31, 2022
Activity
jmerdich commentedon Dec 14, 2022
I'm looking into implementing this but hit an upstream roadblock: lz4/lz4#791. The frame dictionary APIs are still considered unstable (despite their age) and are only available with statically linked copies of lz4. Today, python-lz4 supports both but prefers the dynamically-linked system copy of lz4 if available.
We can fail at runtime still if we are using a dynamically linked copy, but how lz4 was built into the package is pretty opaque to the user.
jonathanunderwood commentedon Dec 14, 2022
Yup - we won't be adding this to the python bindings until the API is stable and public.
jmerdich commentedon Dec 14, 2022
@jonathanunderwood-- do you think having this as experimental only and forcing static links for experimental builds is a good middle ground? This seems like it only isn't stable because there hasn't been a major lz4 release in the last three years.
jonathanunderwood commentedon Dec 14, 2022
I'd consider it if someone submitted a PR and the added complexity in setup.py etc wasn't high and the submitter was prepared to stick around and support it. I have very limited bandwidth to support this package, and this really isn't a high priority compared to the other stuff that is outstanding (fixing/rewriting the frame bindings, Aarch64 builds, type hinting). But honestly, before that, I'd need to see convincing numbers demonstrating it's actually worth it.
jonathanunderwood commentedon Dec 14, 2022
Note also issue #236 means we'll be making changes to setup.py and how compilers and flags are set up.
[-]Don't support dictionary compression[/-][+]Missing support for dictionary compression (unstable, non-public API)[/+]danielhrisca commentedon Jul 22, 2024
Looks like dictionaries are now fully enabled in the LZ4 https://github.com/lz4/lz4/releases/tag/v1.10.0