Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# Tag vX.Y.Z (a release) or vX.Y.Z-rcN, N >= 1 (a release candidate, published as a pre-release that does not become
# "latest"): build and check all four standard flavours and the Windows CUDA 13 SM75 flavour (build.yml), then publish
# "latest"): build and check all four standard flavours, the Windows CUDA 13 SM75 flavour and the Linux CUDA 12 SM70
# flavour (build.yml), then publish
# their archives (the full package and the -system-cuda variant of each) and a SHA256SUMS file as a GitHub release.
# X.Y.Z must equal the MI_VERSION_* of
# include/mossinfer/mossinfer.h and an rc must come before vX.Y.Z (.github/scripts/release_version.py), the tagged
Expand Down
26 changes: 24 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

A native C++/CUDA inference engine for the MOSS-Image text-to-image model (5B DiT, Qwen3.5 text encoder, FLUX.2 VAE), with no PyTorch or Python at run time. It plans memory against the GPU it finds and streams the model when it does not fit.

Models: MOSS-Image Base and SFT only (`OpenMOSS-Team/MOSS-Image-Base`, `OpenMOSS-Team/MOSS-Image-SFT`); other models are refused ([supported models](docs/integration.md#supported-models)).
Models: MOSS-Image Base and SFT, in the official folders and common community formats; other models are refused ([supported models](#supported-models)).

Version 2.1.1. Supported: Linux x86-64 and Windows x64 (MSVC), each with a CUDA 12 and a CUDA 13 build, as a full package and a `-system-cuda` package. Turing (compute capability 7.5, e.g. RTX 2080) is supported by builds configured with `-DMOSSINFER_SM75=ON` (the `windows-msvc-cuda13-sm75` preset packages it): fp16 tensor-core GEMMs replace the bf16 ones, INT8 and INT4 ConvRot files run the INT8 GEMMs, the exact attention and the VAE run sm_75 kernels, all three quality tiers work, and an unset attention resolves to `exact`. See [docs/sm75.md](docs/sm75.md) for what runs, the measured speed on an RTX 2080 Ti and the remaining gaps. Volta (Tesla V100, sm_70) is supported by the CUDA 12 `linux-cuda12-sm70` preset; see [docs/sm70.md](docs/sm70.md) for its kernels, validation and performance. The full support matrix is in [docs/integration.md](docs/integration.md#support-status).
Version 2.1.2. Supported: Linux x86-64 and Windows x64 (MSVC), each with a CUDA 12 and a CUDA 13 build, as a full package and a `-system-cuda` package. Turing (compute capability 7.5, e.g. RTX 2080) is supported by builds configured with `-DMOSSINFER_SM75=ON` (the `windows-msvc-cuda13-sm75` preset packages it): fp16 tensor-core GEMMs replace the bf16 ones, INT8 and INT4 ConvRot files run the INT8 GEMMs, the exact attention and the VAE run sm_75 kernels, all three quality tiers work, and an unset attention resolves to `exact`. See [docs/sm75.md](docs/sm75.md) for what runs, the measured speed on an RTX 2080 Ti and the remaining gaps. Volta (Tesla V100, sm_70) is supported by the CUDA 12 `linux-cuda12-sm70` preset; see [docs/sm70.md](docs/sm70.md) for its kernels, validation and performance. The full support matrix is in [docs/integration.md](docs/integration.md#support-status).

The caller picks the quality tier: `reference` (bf16, exact), `high` (the loaded DiT's GEMMs with 8-bit attention) or `turbo` (a 4-bit DiT). An INT8 DiT with `high` is the everyday choice: 2.5 times the bf16 speed on an RTX 5090, in 40 % less VRAM ([performance](docs/integration.md#performance)). Upgrading from 1.x: [migration](docs/integration.md#migrating-from-1x).

Expand Down Expand Up @@ -42,4 +42,26 @@ CUDA_VISIBLE_DEVICES="" ctest --preset linux-cuda13 -L '^cpu$'

The other presets are `linux-cuda12`, `linux-cuda12-sm70`, `windows-msvc-cuda13`, `windows-msvc-cuda13-sm75` and `windows-msvc-cuda12` ([docs/integration.md](docs/integration.md#getting-the-binaries)). The `windows-msvc-cuda13-sm75` preset builds a full CUDA 13 package with Turing (sm_75) support and names it with the `-sm75` suffix; the `linux-cuda12-sm70` preset adds Volta with a `-sm70` suffix. The standard presets remain Ampere-or-newer.

## Supported models

The engine runs one architecture, MOSS-Image: the 5B DiT, the Qwen3.5-9B text encoder and the FLUX.2 VAE. It identifies a model from its configs and safetensors headers, and refuses anything else with a message naming what it found.

| Component | What loads |
|---|---|
| Model folder | `OpenMOSS-Team/MOSS-Image-Base` and `OpenMOSS-Team/MOSS-Image-SFT` (or the ModelScope copies), and fine-tunes with the same architecture |
| DiT file | INT8 ConvRot (tier `high`), INT4 ConvRot (tier `turbo`), NVFP4 (tier `turbo`, RTX 50 only); GGUF Q8_0, Q6_K, Q5_K, Q4_K, Q4_0, BF16 and F16, converted once on first load |
| Text encoder file | INT8 ConvRot, W4A8 GPTQ |
| VAE file | FLUX.2 VAE, full or small |
| LoRAs and adapters | PEFT, kohya and LyCORIS LoHa LoRAs, distill LoRAs and bundles, the colour and depth control adapter. Runtime LoRAs and control adapters need Ampere or newer |

Not supported: other OpenMOSS models (MOSS-VL, MOVA, MOSS-TTS and others), text encoders of other sizes, torchao and FP8 weights, and MLX, SDNQ, DFloat11 and PyTorch `.bin` / `.pt` files. [docs/integration.md](docs/integration.md#supported-models) lists the files and revisions, how each one loads, and what the engine says when it refuses one.

## Contributors

- [xycld](https://github.com/xycld): the engine
- [CloverIris](https://github.com/CloverIris): Turing (sm_75) support
- [samyycX](https://github.com/samyycX): Volta (sm_70) support

## License

Licensed under Apache-2.0 (`LICENSE`, `NOTICE`, `THIRD_PARTY_NOTICES.md`).
6 changes: 3 additions & 3 deletions docs/integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ The reference documents are:

This page does not repeat them. It explains how the pieces fit together, and which choices the launcher has to make.

Version 2.1.1 (ABI 1, server protocol 2). See [Versioning](#versioning) for what stays compatible, and [Migrating from 1.x](#migrating-from-1x) for what changed.
Version 2.1.2 (ABI 1, server protocol 2). See [Versioning](#versioning) for what stays compatible, and [Migrating from 1.x](#migrating-from-1x) for what changed.

MossInfer is a low-level engine: it runs what the caller asks for, refuses what the hardware or the files cannot run with the reason, and reports what ran. It does not choose for the caller. The launcher picks the tier, the memory mode and the budgets; [Performance](#performance) says what each costs and gives.

Expand Down Expand Up @@ -205,7 +205,7 @@ cmake --build --preset linux-cuda13 -j8
CUDA_VISIBLE_DEVICES="" ctest --preset linux-cuda13 -L '^cpu$'
cmake --install build-linux-cuda13 # -> build-linux-cuda13/install
cmake --install build-linux-cuda13 --component Unspecified --prefix build-linux-cuda13/install-system-cuda
(cd build-linux-cuda13 && cpack --config CPackConfig.cmake) # -> mossinfer-2.1.1-linux-cu13-x86_64.{tar.gz,zip}
(cd build-linux-cuda13 && cpack --config CPackConfig.cmake) # -> mossinfer-2.1.2-linux-cu13-x86_64.{tar.gz,zip}
```

- **glibc:** the archive needs the glibc of the machine that built it, or a newer one. Build on the oldest distribution you want to support. CI uses Ubuntu 22.04, which gives glibc 2.35. A build from a rolling distribution such as Arch will not start on Ubuntu LTS.
Expand Down Expand Up @@ -555,7 +555,7 @@ Attach it to bug reports, together with the server's stderr log and `hello`'s `p
## Versioning

- **Version numbers.**
- `mi_version()` returns the release version (`2.1.1`).
- `mi_version()` returns the release version (`2.1.2`).
- `mi_abi_version()` and `MI_ABI_VERSION` give the binary interface version (1).
- The server's `hello` reports its `protocol_version` (2).
- **Additive changes keep ABI 1:** new functions, new fields at the end of structs, new enum values, new optional server parameters and events.
Expand Down
2 changes: 1 addition & 1 deletion docs/server-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
```
moss-server # speak the protocol on stdin/stdout
moss-server --parent-pid <pid> # the same, and exit when process <pid> (the launcher) exits
moss-server --version # "moss-server 2.1.1 (protocol 2, ABI 1)"
moss-server --version # "moss-server 2.1.2 (protocol 2, ABI 1)"
```

### `--parent-pid`
Expand Down
2 changes: 1 addition & 1 deletion docs/sm75.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ cmake --preset windows-msvc-cuda13-sm75
cmake --build --preset windows-msvc-cuda13-sm75
ctest --preset windows-msvc-cuda13-sm75 -L '^cpu$'
cpack --config build-windows-msvc-cuda13-sm75/CPackConfig.cmake -B build-windows-msvc-cuda13-sm75
# -> build-windows-msvc-cuda13-sm75/mossinfer-2.1.1-windows-cu13-x86_64-sm75.zip
# -> build-windows-msvc-cuda13-sm75/mossinfer-2.1.2-windows-cu13-x86_64-sm75.zip
```

Any other preset takes the option too (`-DMOSSINFER_SM75=ON`), e.g. a Linux build from a custom preset; the ordinary
Expand Down
2 changes: 1 addition & 1 deletion include/mossinfer/mossinfer.h
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ extern "C" {

#define MI_VERSION_MAJOR 2
#define MI_VERSION_MINOR 1
#define MI_VERSION_PATCH 1
#define MI_VERSION_PATCH 2
/* Incremented on every incompatible change of the binary interface (see the conventions above). */
#define MI_ABI_VERSION 1

Expand Down
Loading