From e489edb5c7d124eb92d1f6e019446d634e3b100b Mon Sep 17 00:00:00 2001 From: xycld Date: Sun, 1 Nov 2026 21:07:03 -0500 Subject: [PATCH 1/2] Version 2.1.2: Volta (sm_70) support --- .github/workflows/release.yml | 3 ++- README.md | 2 +- docs/integration.md | 6 +++--- docs/server-protocol.md | 2 +- docs/sm75.md | 2 +- include/mossinfer/mossinfer.h | 2 +- 6 files changed, 9 insertions(+), 8 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 33b66d6..1cb6063 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,5 +1,6 @@ # Tag vX.Y.Z (a release) or vX.Y.Z-rcN, N >= 1 (a release candidate, published as a pre-release that does not become -# "latest"): build and check all four standard flavours and the Windows CUDA 13 SM75 flavour (build.yml), then publish +# "latest"): build and check all four standard flavours, the Windows CUDA 13 SM75 flavour and the Linux CUDA 12 SM70 +# flavour (build.yml), then publish # their archives (the full package and the -system-cuda variant of each) and a SHA256SUMS file as a GitHub release. # X.Y.Z must equal the MI_VERSION_* of # include/mossinfer/mossinfer.h and an rc must come before vX.Y.Z (.github/scripts/release_version.py), the tagged diff --git a/README.md b/README.md index 8674a37..0e2a0e0 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ A native C++/CUDA inference engine for the MOSS-Image text-to-image model (5B Di Models: MOSS-Image Base and SFT only (`OpenMOSS-Team/MOSS-Image-Base`, `OpenMOSS-Team/MOSS-Image-SFT`); other models are refused ([supported models](docs/integration.md#supported-models)). -Version 2.1.1. Supported: Linux x86-64 and Windows x64 (MSVC), each with a CUDA 12 and a CUDA 13 build, as a full package and a `-system-cuda` package. Turing (compute capability 7.5, e.g. RTX 2080) is supported by builds configured with `-DMOSSINFER_SM75=ON` (the `windows-msvc-cuda13-sm75` preset packages it): fp16 tensor-core GEMMs replace the bf16 ones, INT8 and INT4 ConvRot files run the INT8 GEMMs, the exact attention and the VAE run sm_75 kernels, all three quality tiers work, and an unset attention resolves to `exact`. See [docs/sm75.md](docs/sm75.md) for what runs, the measured speed on an RTX 2080 Ti and the remaining gaps. Volta (Tesla V100, sm_70) is supported by the CUDA 12 `linux-cuda12-sm70` preset; see [docs/sm70.md](docs/sm70.md) for its kernels, validation and performance. The full support matrix is in [docs/integration.md](docs/integration.md#support-status). +Version 2.1.2. Supported: Linux x86-64 and Windows x64 (MSVC), each with a CUDA 12 and a CUDA 13 build, as a full package and a `-system-cuda` package. Turing (compute capability 7.5, e.g. RTX 2080) is supported by builds configured with `-DMOSSINFER_SM75=ON` (the `windows-msvc-cuda13-sm75` preset packages it): fp16 tensor-core GEMMs replace the bf16 ones, INT8 and INT4 ConvRot files run the INT8 GEMMs, the exact attention and the VAE run sm_75 kernels, all three quality tiers work, and an unset attention resolves to `exact`. See [docs/sm75.md](docs/sm75.md) for what runs, the measured speed on an RTX 2080 Ti and the remaining gaps. Volta (Tesla V100, sm_70) is supported by the CUDA 12 `linux-cuda12-sm70` preset; see [docs/sm70.md](docs/sm70.md) for its kernels, validation and performance. The full support matrix is in [docs/integration.md](docs/integration.md#support-status). The caller picks the quality tier: `reference` (bf16, exact), `high` (the loaded DiT's GEMMs with 8-bit attention) or `turbo` (a 4-bit DiT). An INT8 DiT with `high` is the everyday choice: 2.5 times the bf16 speed on an RTX 5090, in 40 % less VRAM ([performance](docs/integration.md#performance)). Upgrading from 1.x: [migration](docs/integration.md#migrating-from-1x). diff --git a/docs/integration.md b/docs/integration.md index 0c401b7..4bc856b 100644 --- a/docs/integration.md +++ b/docs/integration.md @@ -8,7 +8,7 @@ The reference documents are: This page does not repeat them. It explains how the pieces fit together, and which choices the launcher has to make. -Version 2.1.1 (ABI 1, server protocol 2). See [Versioning](#versioning) for what stays compatible, and [Migrating from 1.x](#migrating-from-1x) for what changed. +Version 2.1.2 (ABI 1, server protocol 2). See [Versioning](#versioning) for what stays compatible, and [Migrating from 1.x](#migrating-from-1x) for what changed. MossInfer is a low-level engine: it runs what the caller asks for, refuses what the hardware or the files cannot run with the reason, and reports what ran. It does not choose for the caller. The launcher picks the tier, the memory mode and the budgets; [Performance](#performance) says what each costs and gives. @@ -205,7 +205,7 @@ cmake --build --preset linux-cuda13 -j8 CUDA_VISIBLE_DEVICES="" ctest --preset linux-cuda13 -L '^cpu$' cmake --install build-linux-cuda13 # -> build-linux-cuda13/install cmake --install build-linux-cuda13 --component Unspecified --prefix build-linux-cuda13/install-system-cuda -(cd build-linux-cuda13 && cpack --config CPackConfig.cmake) # -> mossinfer-2.1.1-linux-cu13-x86_64.{tar.gz,zip} +(cd build-linux-cuda13 && cpack --config CPackConfig.cmake) # -> mossinfer-2.1.2-linux-cu13-x86_64.{tar.gz,zip} ``` - **glibc:** the archive needs the glibc of the machine that built it, or a newer one. Build on the oldest distribution you want to support. CI uses Ubuntu 22.04, which gives glibc 2.35. A build from a rolling distribution such as Arch will not start on Ubuntu LTS. @@ -555,7 +555,7 @@ Attach it to bug reports, together with the server's stderr log and `hello`'s `p ## Versioning - **Version numbers.** - - `mi_version()` returns the release version (`2.1.1`). + - `mi_version()` returns the release version (`2.1.2`). - `mi_abi_version()` and `MI_ABI_VERSION` give the binary interface version (1). - The server's `hello` reports its `protocol_version` (2). - **Additive changes keep ABI 1:** new functions, new fields at the end of structs, new enum values, new optional server parameters and events. diff --git a/docs/server-protocol.md b/docs/server-protocol.md index e34356c..3dc60b1 100644 --- a/docs/server-protocol.md +++ b/docs/server-protocol.md @@ -8,7 +8,7 @@ ``` moss-server # speak the protocol on stdin/stdout moss-server --parent-pid # the same, and exit when process (the launcher) exits -moss-server --version # "moss-server 2.1.1 (protocol 2, ABI 1)" +moss-server --version # "moss-server 2.1.2 (protocol 2, ABI 1)" ``` ### `--parent-pid` diff --git a/docs/sm75.md b/docs/sm75.md index 5d8e718..1f7f413 100644 --- a/docs/sm75.md +++ b/docs/sm75.md @@ -17,7 +17,7 @@ cmake --preset windows-msvc-cuda13-sm75 cmake --build --preset windows-msvc-cuda13-sm75 ctest --preset windows-msvc-cuda13-sm75 -L '^cpu$' cpack --config build-windows-msvc-cuda13-sm75/CPackConfig.cmake -B build-windows-msvc-cuda13-sm75 -# -> build-windows-msvc-cuda13-sm75/mossinfer-2.1.1-windows-cu13-x86_64-sm75.zip +# -> build-windows-msvc-cuda13-sm75/mossinfer-2.1.2-windows-cu13-x86_64-sm75.zip ``` Any other preset takes the option too (`-DMOSSINFER_SM75=ON`), e.g. a Linux build from a custom preset; the ordinary diff --git a/include/mossinfer/mossinfer.h b/include/mossinfer/mossinfer.h index 2fba555..d4f14ac 100644 --- a/include/mossinfer/mossinfer.h +++ b/include/mossinfer/mossinfer.h @@ -75,7 +75,7 @@ extern "C" { #define MI_VERSION_MAJOR 2 #define MI_VERSION_MINOR 1 -#define MI_VERSION_PATCH 1 +#define MI_VERSION_PATCH 2 /* Incremented on every incompatible change of the binary interface (see the conventions above). */ #define MI_ABI_VERSION 1 From 222be934ba1f97adef52bbd56d2a835620a6cdf2 Mon Sep 17 00:00:00 2001 From: xycld Date: Sun, 1 Nov 2026 22:17:51 -0500 Subject: [PATCH 2/2] README: which models run, and the contributors --- README.md | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 0e2a0e0..ccfc4fb 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ A native C++/CUDA inference engine for the MOSS-Image text-to-image model (5B DiT, Qwen3.5 text encoder, FLUX.2 VAE), with no PyTorch or Python at run time. It plans memory against the GPU it finds and streams the model when it does not fit. -Models: MOSS-Image Base and SFT only (`OpenMOSS-Team/MOSS-Image-Base`, `OpenMOSS-Team/MOSS-Image-SFT`); other models are refused ([supported models](docs/integration.md#supported-models)). +Models: MOSS-Image Base and SFT, in the official folders and common community formats; other models are refused ([supported models](#supported-models)). Version 2.1.2. Supported: Linux x86-64 and Windows x64 (MSVC), each with a CUDA 12 and a CUDA 13 build, as a full package and a `-system-cuda` package. Turing (compute capability 7.5, e.g. RTX 2080) is supported by builds configured with `-DMOSSINFER_SM75=ON` (the `windows-msvc-cuda13-sm75` preset packages it): fp16 tensor-core GEMMs replace the bf16 ones, INT8 and INT4 ConvRot files run the INT8 GEMMs, the exact attention and the VAE run sm_75 kernels, all three quality tiers work, and an unset attention resolves to `exact`. See [docs/sm75.md](docs/sm75.md) for what runs, the measured speed on an RTX 2080 Ti and the remaining gaps. Volta (Tesla V100, sm_70) is supported by the CUDA 12 `linux-cuda12-sm70` preset; see [docs/sm70.md](docs/sm70.md) for its kernels, validation and performance. The full support matrix is in [docs/integration.md](docs/integration.md#support-status). @@ -42,4 +42,26 @@ CUDA_VISIBLE_DEVICES="" ctest --preset linux-cuda13 -L '^cpu$' The other presets are `linux-cuda12`, `linux-cuda12-sm70`, `windows-msvc-cuda13`, `windows-msvc-cuda13-sm75` and `windows-msvc-cuda12` ([docs/integration.md](docs/integration.md#getting-the-binaries)). The `windows-msvc-cuda13-sm75` preset builds a full CUDA 13 package with Turing (sm_75) support and names it with the `-sm75` suffix; the `linux-cuda12-sm70` preset adds Volta with a `-sm70` suffix. The standard presets remain Ampere-or-newer. +## Supported models + +The engine runs one architecture, MOSS-Image: the 5B DiT, the Qwen3.5-9B text encoder and the FLUX.2 VAE. It identifies a model from its configs and safetensors headers, and refuses anything else with a message naming what it found. + +| Component | What loads | +|---|---| +| Model folder | `OpenMOSS-Team/MOSS-Image-Base` and `OpenMOSS-Team/MOSS-Image-SFT` (or the ModelScope copies), and fine-tunes with the same architecture | +| DiT file | INT8 ConvRot (tier `high`), INT4 ConvRot (tier `turbo`), NVFP4 (tier `turbo`, RTX 50 only); GGUF Q8_0, Q6_K, Q5_K, Q4_K, Q4_0, BF16 and F16, converted once on first load | +| Text encoder file | INT8 ConvRot, W4A8 GPTQ | +| VAE file | FLUX.2 VAE, full or small | +| LoRAs and adapters | PEFT, kohya and LyCORIS LoHa LoRAs, distill LoRAs and bundles, the colour and depth control adapter. Runtime LoRAs and control adapters need Ampere or newer | + +Not supported: other OpenMOSS models (MOSS-VL, MOVA, MOSS-TTS and others), text encoders of other sizes, torchao and FP8 weights, and MLX, SDNQ, DFloat11 and PyTorch `.bin` / `.pt` files. [docs/integration.md](docs/integration.md#supported-models) lists the files and revisions, how each one loads, and what the engine says when it refuses one. + +## Contributors + +- [xycld](https://github.com/xycld): the engine +- [CloverIris](https://github.com/CloverIris): Turing (sm_75) support +- [samyycX](https://github.com/samyycX): Volta (sm_70) support + +## License + Licensed under Apache-2.0 (`LICENSE`, `NOTICE`, `THIRD_PARTY_NOTICES.md`).