Skip to content

New compressions - #3

Open
dirkboye wants to merge 8 commits into
mainfrom
new-compressions
Open

dirkboye wants to merge 8 commits into
mainfrom
new-compressions

Conversation

@dirkboye

Copy link
Copy Markdown
Collaborator

No description provided.

macos-11, macos-12, macos-13, ubuntu-20.04 and windows-2019 are no longer
provided by GitHub, so their jobs stay queued and never run.
GoogleTest 1.10.0 does not compile with recent compilers (e.g. GCC 15 and
Clang 21 on Ubuntu 26.04) because of a missing <cstdint> include. 1.16.0
is the last release that supports C++14 (MSVC's default); it requires at
least C++14, so override the -std=c++0x default of Bazel's Unix toolchain.
Test on ubuntu-24.04, ubuntu-26.04, macos-15 (arm64), macos-15-intel,
macos-26 (arm64), windows-2025 and windows-11-arm (CMake and Python only)
in addition to ubuntu-22.04 and windows-2022. Exclude Python versions that
actions/setup-python does not provide for these images.
The error message for an invalid keyword argument changed in Python 3.13.
Bazel 9 no longer supports WORKSPACE, so the repository becomes the Bazel
module `dectris_compression`. The Bazel build does not use the git
submodules anymore: lz4 and zstd come from the Bazel Central Registry,
bitshuffle and TRPX are fetched with http_archive at the commits of the
submodules (the TRPX patches are applied by http_archive). A source archive
without submodules can therefore be used directly as a module dependency,
without a BUILD file in the depending repository.

The header is exposed as "dectris_compression/src/compression.h". The
sources include the third party headers without directory prefixes, which
CMake and setup.py now provide as include directories.

Builds and tests use the hermetic LLVM 22.1.6 toolchain (toolchains_llvm,
development dependency) on Linux and macOS (arm64), MSVC on Windows. The
TRPX headers are external includes (feature external_include_paths), which
suppresses their warnings also with MSVC. Failing tests print their log.
Not supported are macOS x86_64 (no LLVM release), macOS 15 (the LLVM 22
libc++ headers require a newer system libc++) and Ubuntu 26.04 (ld.lld
requires libxml2.so.2). The sanitizers run on Linux only, since the test
binaries do not find the sanitizer dylibs on macOS.

A new TRPX patch fixes advancing the end iterator in
XML_element::add_attribute, which the MSVC debug runtime reports when the
tests write TRPX data.

32 bit Windows (x86) is no longer built and tested. The TRPX patch for
32 bit platforms is dropped; the patch fixing `1ul << B` shifts is kept,
since `unsigned long` has 32 bits also on 64 bit Windows.

CMake applies the TRPX patches with apply_patches.py (as setup.py) instead
of `patch`, so it requires Python 3. `patch` failed on Windows because of
CRLF line endings (git core.autocrlf): CMake's file(WRITE) writes CRLF on
Windows, and the patch of the runners strips the CRs of the patch only.

The workflow uses actions/checkout@v7 and actions/setup-python@v7, which
run on Node.js 24 (Node.js 20 is deprecated on the runners).

All CI matrices use `fail-fast: false`, so a failing job does not cancel
the others.

After all builds and tests passed, a new job creates the source
distribution for PyPI, checks it with twine and uploads it as the
workflow artifact `sdist`.
Adds compression to the C API, which so far only supported decompression:

- compression_compress_buffer() and compression_compress_bound() compress
  a whole buffer with COMPRESSION_BSZSTD or COMPRESSION_TRPX.
- compression_stream_encoder_*() and compression_stream_bound() compress
  a stream block by block with COMPRESSION_BSZSTD. The encoder reuses its
  Zstandard context and bitshuffle buffers for all blocks and streams.

COMPRESSION_BSZSTD writes the format of the bitshuffle HDF5 filter with
the default block size of bitshuffle unless a block size is given, and
Zstandard compression level 1. COMPRESSION_TRPX writes a single TRPX
stream with one frame, as written by Terse.

CMake and setup.py now also build the Zstandard compressor. The Python
module does not provide compression.
@dirkboye
dirkboye requested a review from petm-dec October 19, 2026 06:01
@dirkboye dirkboye self-assigned this Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant