Skip to content

[Proposal / Benchmark] 100% Lossless (Bit-Exact) BF16 Bit-Stream Compression & CUDA Decompression Engine #1081

Open

Description

@wiobe

Hi Gemma.cpp team,

I've been working on low-level memory and PCIe transfer optimizations for BFloat16 (BF16) LLM weights, and recently open-sourced a standalone C++17 / NVIDIA CUDA prototype called ZeroLoss-BF16-Engine (ZBF16).

Since gemma.cpp focuses heavily on lightweight, high-performance C++ inference and compressed weight formats (like SFP), I wanted to share our lossless BF16 compression benchmarks in case a zero-degradation bit-stream loader is of interest for BF16 checkpoints:

Approach

  1. Plane Splitting & Sign-Bit Rotation: Separates the volatile 8-bit mantissa from the sign+exponent plane and rotates the sign bit to the LSB position to align positive/negative weight magnitudes.
  2. Pivot XOR & Prefix-Free Bit-Stream: Differentiates against the statistical distribution pivot (0x79) and packs 2-bit pairs without framing headers.
  3. Register-Cached Parallel Decoding: Uses independent 64-element blocks with 32-bit register bit-caches for high-throughput parallel reconstruction.

Hardware Benchmarks (NVIDIA RTX 3090, C++17 / CUDA 13.2)

  • Net Lossless Compression: Reduced a 14.19 GB BF16 model (Qwen2.5-VL-7B) to 12.23 GB (2,002 MB net savings; 27.55% reduction on the upper exponent plane).
  • Bit-Exact Verification: Verified 100% SHA-256 checksum identity (0 bit errors across 7.61 billion parameters) after full pack/unpack cycles.
  • Parallel Decompression Latency: 0.3358 ms for a 4.19 MB tensor slice across 65,536 threads.
  • Live Inference Integration (llama.cpp): Intercepted 150+ BF16 tensors during load with 57.9 tokens/sec generation speed (matching the 58.0 t/s uncompressed baseline).

Full C++/CUDA source code, integration patch, and verification logs are available under AGPLv3 here:

👉 https://github.com/wiobe/ZeroLoss-BF16-Engine

Would love to hear your thoughts on whether a similar lossless exponent-stream approach could complement gemma.cpp's weight loading pipeline!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions