You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've been working on low-level memory and PCIe transfer optimizations for BFloat16 (BF16) LLM weights, and recently open-sourced a standalone C++17 / NVIDIA CUDA prototype called ZeroLoss-BF16-Engine (ZBF16).
Since gemma.cpp focuses heavily on lightweight, high-performance C++ inference and compressed weight formats (like SFP), I wanted to share our lossless BF16 compression benchmarks in case a zero-degradation bit-stream loader is of interest for BF16 checkpoints:
Approach
Plane Splitting & Sign-Bit Rotation: Separates the volatile 8-bit mantissa from the sign+exponent plane and rotates the sign bit to the LSB position to align positive/negative weight magnitudes.
Pivot XOR & Prefix-Free Bit-Stream: Differentiates against the statistical distribution pivot (0x79) and packs 2-bit pairs without framing headers.
Register-Cached Parallel Decoding: Uses independent 64-element blocks with 32-bit register bit-caches for high-throughput parallel reconstruction.
Hardware Benchmarks (NVIDIA RTX 3090, C++17 / CUDA 13.2)
Net Lossless Compression: Reduced a 14.19 GB BF16 model (Qwen2.5-VL-7B) to 12.23 GB (2,002 MB net savings; 27.55% reduction on the upper exponent plane).
Bit-Exact Verification: Verified 100% SHA-256 checksum identity (0 bit errors across 7.61 billion parameters) after full pack/unpack cycles.
Parallel Decompression Latency:0.3358 ms for a 4.19 MB tensor slice across 65,536 threads.
Live Inference Integration (llama.cpp): Intercepted 150+ BF16 tensors during load with 57.9 tokens/sec generation speed (matching the 58.0 t/s uncompressed baseline).
Full C++/CUDA source code, integration patch, and verification logs are available under AGPLv3 here:
Activity