Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ A native C++/CUDA inference engine for the MOSS-Image text-to-image model (5B Di

Models: MOSS-Image Base and SFT, in the official folders and common community formats; other models are refused ([supported models](#supported-models)).

Version 2.1.2. Supported: Linux x86-64 and Windows x64 (MSVC), each with a CUDA 12 and a CUDA 13 build, as a full package and a `-system-cuda` package. Turing (compute capability 7.5, e.g. RTX 2080) is supported by builds configured with `-DMOSSINFER_SM75=ON` (the `windows-msvc-cuda13-sm75` preset packages it): fp16 tensor-core GEMMs replace the bf16 ones, INT8 and INT4 ConvRot files run the INT8 GEMMs, the exact attention and the VAE run sm_75 kernels, all three quality tiers work, and an unset attention resolves to `exact`. See [docs/sm75.md](docs/sm75.md) for what runs, the measured speed on an RTX 2080 Ti and the remaining gaps. Volta (Tesla V100, sm_70) is supported by the CUDA 12 `linux-cuda12-sm70` preset; see [docs/sm70.md](docs/sm70.md) for its kernels, validation and performance. Other GPUs (AMD, Intel, older NVIDIA) can run txt2img at the reference tier through `moss-infer-vulkan`, a separate and slower Vulkan program in its own package ([docs/vulkan.md](docs/vulkan.md)); on Linux, `moss-infer-hip`, a separate ROCm 7 program, runs it on AMD GPUs: built for gfx1100, gfx1200 and gfx1201 (Radeon RX 7900 and RX 9000), validated on gfx1200 (RX 9060 XT) ([docs/hip.md](docs/hip.md)). The full support matrix is in [docs/integration.md](docs/integration.md#support-status).
Version 2.2.0. Supported: Linux x86-64 and Windows x64 (MSVC), each with a CUDA 12 and a CUDA 13 build, as a full package and a `-system-cuda` package. Turing (compute capability 7.5, e.g. RTX 2080) is supported by builds configured with `-DMOSSINFER_SM75=ON` (the `windows-msvc-cuda13-sm75` preset packages it): fp16 tensor-core GEMMs replace the bf16 ones, INT8 and INT4 ConvRot files run the INT8 GEMMs, the exact attention and the VAE run sm_75 kernels, all three quality tiers work, and an unset attention resolves to `exact`. See [docs/sm75.md](docs/sm75.md) for what runs, the measured speed on an RTX 2080 Ti and the remaining gaps. Volta (Tesla V100, sm_70) is supported by the CUDA 12 `linux-cuda12-sm70` preset; see [docs/sm70.md](docs/sm70.md) for its kernels, validation and performance. Other GPUs (AMD, Intel, older NVIDIA) can run txt2img at the reference tier through `moss-infer-vulkan`, a separate and slower Vulkan program in its own package ([docs/vulkan.md](docs/vulkan.md)); on Linux, `moss-infer-hip`, a separate ROCm 7 program, runs it on AMD GPUs: built for gfx1100, gfx1200 and gfx1201 (Radeon RX 7900 and RX 9000), validated on gfx1200 (RX 9060 XT) ([docs/hip.md](docs/hip.md)). The full support matrix is in [docs/integration.md](docs/integration.md#support-status).

The caller picks the quality tier: `reference` (bf16, exact), `high` (the loaded DiT's GEMMs with 8-bit attention) or `turbo` (a 4-bit DiT). An INT8 DiT with `high` is the everyday choice: 2.5 times the bf16 speed on an RTX 5090, in 40 % less VRAM ([performance](docs/integration.md#performance)). Upgrading from 1.x: [migration](docs/integration.md#migrating-from-1x).

Expand Down
6 changes: 3 additions & 3 deletions docs/integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ The reference documents are:

This page does not repeat them. It explains how the pieces fit together, and which choices the launcher has to make.

Version 2.1.2 (ABI 1, server protocol 2). See [Versioning](#versioning) for what stays compatible, and [Migrating from 1.x](#migrating-from-1x) for what changed.
Version 2.2.0 (ABI 1, server protocol 2). See [Versioning](#versioning) for what stays compatible, and [Migrating from 1.x](#migrating-from-1x) for what changed.

MossInfer is a low-level engine: it runs what the caller asks for, refuses what the hardware or the files cannot run with the reason, and reports what ran. It does not choose for the caller. The launcher picks the tier, the memory mode and the budgets; [Performance](#performance) says what each costs and gives.

Expand Down Expand Up @@ -205,7 +205,7 @@ cmake --build --preset linux-cuda13 -j8
CUDA_VISIBLE_DEVICES="" ctest --preset linux-cuda13 -L '^cpu$'
cmake --install build-linux-cuda13 # -> build-linux-cuda13/install
cmake --install build-linux-cuda13 --component Unspecified --prefix build-linux-cuda13/install-system-cuda
(cd build-linux-cuda13 && cpack --config CPackConfig.cmake) # -> mossinfer-2.1.2-linux-cu13-x86_64.{tar.gz,zip}
(cd build-linux-cuda13 && cpack --config CPackConfig.cmake) # -> mossinfer-2.2.0-linux-cu13-x86_64.{tar.gz,zip}
```

- **glibc:** the archive needs the glibc of the machine that built it, or a newer one. Build on the oldest distribution you want to support. CI uses Ubuntu 22.04, which gives glibc 2.35. A build from a rolling distribution such as Arch will not start on Ubuntu LTS.
Expand Down Expand Up @@ -555,7 +555,7 @@ Attach it to bug reports, together with the server's stderr log and `hello`'s `p
## Versioning

- **Version numbers.**
- `mi_version()` returns the release version (`2.1.2`).
- `mi_version()` returns the release version (`2.2.0`).
- `mi_abi_version()` and `MI_ABI_VERSION` give the binary interface version (1).
- The server's `hello` reports its `protocol_version` (2).
- **Additive changes keep ABI 1:** new functions, new fields at the end of structs, new enum values, new optional server parameters and events.
Expand Down
2 changes: 1 addition & 1 deletion docs/server-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
```
moss-server # speak the protocol on stdin/stdout
moss-server --parent-pid <pid> # the same, and exit when process <pid> (the launcher) exits
moss-server --version # "moss-server 2.1.2 (protocol 2, ABI 1)"
moss-server --version # "moss-server 2.2.0 (protocol 2, ABI 1)"
```

### `--parent-pid`
Expand Down
2 changes: 1 addition & 1 deletion docs/sm75.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ cmake --preset windows-msvc-cuda13-sm75
cmake --build --preset windows-msvc-cuda13-sm75
ctest --preset windows-msvc-cuda13-sm75 -L '^cpu$'
cpack --config build-windows-msvc-cuda13-sm75/CPackConfig.cmake -B build-windows-msvc-cuda13-sm75
# -> build-windows-msvc-cuda13-sm75/mossinfer-2.1.2-windows-cu13-x86_64-sm75.zip
# -> build-windows-msvc-cuda13-sm75/mossinfer-2.2.0-windows-cu13-x86_64-sm75.zip
```

Any other preset takes the option too (`-DMOSSINFER_SM75=ON`), e.g. a Linux build from a custom preset; the ordinary
Expand Down
3 changes: 2 additions & 1 deletion docs/vulkan.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ driver's watchdog on a slow GPU.

## Accuracy

Against the official PyTorch pipeline's golden data at 1024x1024 (`tests/golden/g_fridge_1024`), on the `coop` path:
Against the official PyTorch pipeline's golden data at 1024x1024 (`tests/golden/g_fridge_1024`), on the `coop` path. All
figures on this page come from an RTX 4090 Laptop GPU (every path); CI runs the kernel checks on lavapipe (`plain`).

| Stage | Result |
|---|---|
Expand Down
4 changes: 2 additions & 2 deletions include/mossinfer/mossinfer.h
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,8 @@ extern "C" {
/* ------------------------------------------------------------------------------------------------ version */

#define MI_VERSION_MAJOR 2
#define MI_VERSION_MINOR 1
#define MI_VERSION_PATCH 2
#define MI_VERSION_MINOR 2
#define MI_VERSION_PATCH 0
/* Incremented on every incompatible change of the binary interface (see the conventions above). */
#define MI_ABI_VERSION 1

Expand Down
Loading