From 29b679e1b40a3a6a66de55476fc864c3dd656082 Mon Sep 17 00:00:00 2001 From: xycld Date: Sat, 31 Oct 2026 12:50:51 -0400 Subject: [PATCH] Version 2.1.1: Turing (sm_75) support Bumps MI_VERSION_PATCH and the version strings in the docs. docs/sm75.md and docs/integration.md carry the step times and peaks measured on an RTX 2080 Ti with the merged follow-ups. docs/c-api.md and docs/server-protocol.md: the "2.x" labels become 2.1.0. The 408-byte mi_gen_params, reference images, LoRA placement, mi_check and load's parameter check all shipped in the v2.1.0 tag; the 376-byte layout was 2.0.0's. --- README.md | 4 ++-- docs/c-api.md | 6 +++--- docs/integration.md | 14 +++++++------- docs/server-protocol.md | 4 ++-- docs/sm75.md | 33 +++++++++++++++++---------------- include/mossinfer/mossinfer.h | 2 +- 6 files changed, 32 insertions(+), 31 deletions(-) diff --git a/README.md b/README.md index 674a370..e98a442 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ A native C++/CUDA inference engine for the MOSS-Image text-to-image model (5B Di Models: MOSS-Image Base and SFT only (`OpenMOSS-Team/MOSS-Image-Base`, `OpenMOSS-Team/MOSS-Image-SFT`); other models are refused ([supported models](docs/integration.md#supported-models)). -Version 2.1.0. Supported: Linux x86-64 and Windows x64 (MSVC), each with a CUDA 12 and a CUDA 13 build, as a full package and a `-system-cuda` package. Turing (compute capability 7.5, e.g. RTX 2080) is supported by builds configured with `-DMOSSINFER_SM75=ON` (the `windows-msvc-cuda13-sm75` preset packages it): fp16 tensor-core GEMMs replace the bf16 ones, INT8 and INT4 ConvRot files run the INT8 GEMMs, the exact attention and the VAE run sm_75 kernels, all three quality tiers work, and an unset attention resolves to `exact`. See [docs/sm75.md](docs/sm75.md) for what runs, the measured speed on an RTX 2080 and the remaining gaps. The full support matrix is in [docs/integration.md](docs/integration.md#support-status). +Version 2.1.1. Supported: Linux x86-64 and Windows x64 (MSVC), each with a CUDA 12 and a CUDA 13 build, as a full package and a `-system-cuda` package. Turing (compute capability 7.5, e.g. RTX 2080) is supported by builds configured with `-DMOSSINFER_SM75=ON` (the `windows-msvc-cuda13-sm75` preset packages it): fp16 tensor-core GEMMs replace the bf16 ones, INT8 and INT4 ConvRot files run the INT8 GEMMs, the exact attention and the VAE run sm_75 kernels, all three quality tiers work, and an unset attention resolves to `exact`. See [docs/sm75.md](docs/sm75.md) for what runs, the measured speed on an RTX 2080 Ti and the remaining gaps. The full support matrix is in [docs/integration.md](docs/integration.md#support-status). The caller picks the quality tier: `reference` (bf16, exact), `high` (the loaded DiT's GEMMs with 8-bit attention) or `turbo` (a 4-bit DiT). An INT8 DiT with `high` is the everyday choice: 2.5 times the bf16 speed on an RTX 5090, in 40 % less VRAM ([performance](docs/integration.md#performance)). Upgrading from 1.x: [migration](docs/integration.md#migrating-from-1x). @@ -14,7 +14,7 @@ moss-infer generate --model MOSS-Image/sft --dit moss_image_sft_int8_convrot.saf ``` - [docs/integration.md](docs/integration.md): building the engine into an application. Start here. -- [docs/sm75.md](docs/sm75.md): Turing (compute capability 7.5) support: build and package, what runs, measured speed on an RTX 2080, limitations. +- [docs/sm75.md](docs/sm75.md): Turing (compute capability 7.5) support: build and package, what runs, measured speed on an RTX 2080 Ti, limitations. - [docs/c-api.md](docs/c-api.md): the C library, `include/mossinfer/mossinfer.h`. - [docs/server-protocol.md](docs/server-protocol.md): `moss-server`, the JSON-lines sidecar process. diff --git a/docs/c-api.md b/docs/c-api.md index 748c956..570958c 100644 --- a/docs/c-api.md +++ b/docs/c-api.md @@ -682,7 +682,7 @@ A versioned struct only grows at the end, so a program built against an older he | Struct | Accepted `struct_size` | |---|---| | `mi_engine_params` | 80 (the 0.1.0 size, up to `_reserved1`), 120 (an earlier 0.2.0 header, up to `_reserved2`), 136 (0.2.0 and 1.0.0, up to `distill_dir`), 160 (2.0.0) | -| `mi_gen_params` | 168 (the 0.1.0 size, up to `user_data`), 240 (an earlier 0.2.0 header, up to `_reserved2`), 320 (0.2.0 and 1.0.0, up to `distill_dir`), 376 (2.0.0 and 2.1.0, up to `step_cache_threshold`), 408 (2.x, with the reference images and `lora_placement`) | +| `mi_gen_params` | 168 (the 0.1.0 size, up to `user_data`), 240 (an earlier 0.2.0 header, up to `_reserved2`), 320 (0.2.0 and 1.0.0, up to `distill_dir`), 376 (2.0.0, up to `step_cache_threshold`), 408 (2.1.0, with the reference images and `lora_placement`) | | `mi_timings` (output) | 104 (the 0.1.0 size, up to `vae_load_s`), 128 (0.2.0 and 1.0.0, up to `image_ops_s`), 144 (2.0.0) | | `mi_memory_stats` (output) | 56 (1.0.0, up to `host_private_peak_bytes`), 80 (2.0.0) | | `mi_probe_info` (output) | 576 (the first 0.2.0 size, up to `_reserved0`), 1624 (0.2.0 with the cuBLASLt fields) | @@ -758,8 +758,8 @@ History, all with `MI_ABI_VERSION` 1: - 2.0.0, file inspection: `mi_inspect_file()`, the opaque `mi_file_report` with `mi_file_report_info()`, `_text()`, `_control_kind()`, `_tier_reason()` and `_free()`, `mi_file_info`, the enums `mi_file_kind` and `mi_file_text_kind`, and `MI_CAP_INSPECT` were added. - 2.0.0, explicit tiers (a major version: callers must act, the layout is unchanged): `MI_TIER_AUTO`, the zero default of `mi_engine_params.tier`, is refused by `mi_engine_create()`. Quality certificates are gone from the engine; their fields stay in place, reserved: `certificate_dirs` / `certificate_dirs_count` are ignored, `MI_CAP_CERTIFICATE_DIRS` is never set, `mi_tier_info.certified` and `run_certified` are 0, `MI_TEXT_CERTIFICATE` is `""`. The report's `models.dit` has `tier` instead of `tier_requested`, `tier_effective` and `certified`, and its `quality` object no longer has `reason`, `certificate` or the models' content hashes. The SONAME stays `libmossinfer.so.1` (`MI_ABI_VERSION` 1): no symbol, field, offset or enum value moved, so a 1.x binary loads and fails only where it relied on `AUTO`, with a clear `MI_ERR_INVALID_ARGUMENT`. Migration: [integration.md](integration.md#migrating-from-1x). - 2.1.0, the component API (additive): the structs `mi_tensor`, `mi_session_params`, `mi_text_params`, `mi_conditioning_info`, `mi_run_row`, `mi_run_params`, `mi_run_info`, `mi_step_params` and `mi_step_info`, the enums `mi_memory`, `mi_dtype`, `mi_guidance` and `mi_step_output`, the opaque `mi_conditioning`, 25 functions, `MI_CAP_COMPONENTS` (0x40000) and `MI_TEXT_COMPONENT_NOTE` (18) were added. `mi_engine_free()` now cancels and waits for a job running on another thread instead of leaving that race undefined. -- 2.x, the check (additive): `mi_check()`, the opaque `mi_check_report` with `mi_check_report_info()`, `_text()` and `_free()`, `mi_check_info` (136 bytes), the enums `mi_check_stage` and `mi_check_text_kind`, and `MI_CAP_CHECK` (0x100000) were added; the plan callback's JSON gained `"estimate"` as its last member. -- 2.x (the release after 2.1.0), reference images and LoRA placement (additive): `mi_gen_params` grew to 408 bytes (`reference_images`, `reference_images_count`, `reference_max_edge`, `_reserved5`, `lora_placement`, `_reserved6`), as trailing fields; the 376-byte layout stays accepted (no reference images, the default placement). `mi_reference_image` (24 bytes), `mi_lora_placement`, `MI_CAP_REFERENCE_IMAGES` (0x80000), `MI_TEXT_REFERENCES` (19) and `MI_MESSAGE_REFERENCE_WARNING` (5) were added. +- 2.1.0, the check (additive): `mi_check()`, the opaque `mi_check_report` with `mi_check_report_info()`, `_text()` and `_free()`, `mi_check_info` (136 bytes), the enums `mi_check_stage` and `mi_check_text_kind`, and `MI_CAP_CHECK` (0x100000) were added; the plan callback's JSON gained `"estimate"` as its last member. +- 2.1.0, reference images and LoRA placement (additive): `mi_gen_params` grew to 408 bytes (`reference_images`, `reference_images_count`, `reference_max_edge`, `_reserved5`, `lora_placement`, `_reserved6`), as trailing fields; the 376-byte layout stays accepted (no reference images, the default placement). `mi_reference_image` (24 bytes), `mi_lora_placement`, `MI_CAP_REFERENCE_IMAGES` (0x80000), `MI_TEXT_REFERENCES` (19) and `MI_MESSAGE_REFERENCE_WARNING` (5) were added. ## Symbol visibility diff --git a/docs/integration.md b/docs/integration.md index 9c2f5e8..1e8b6a2 100644 --- a/docs/integration.md +++ b/docs/integration.md @@ -8,7 +8,7 @@ The reference documents are: This page does not repeat them. It explains how the pieces fit together, and which choices the launcher has to make. -Version 2.1.0 (ABI 1, server protocol 2). See [Versioning](#versioning) for what stays compatible, and [Migrating from 1.x](#migrating-from-1x) for what changed. +Version 2.1.1 (ABI 1, server protocol 2). See [Versioning](#versioning) for what stays compatible, and [Migrating from 1.x](#migrating-from-1x) for what changed. MossInfer is a low-level engine: it runs what the caller asks for, refuses what the hardware or the files cannot run with the reason, and reports what ran. It does not choose for the caller. The launcher picks the tier, the memory mode and the budgets; [Performance](#performance) says what each costs and gives. @@ -205,7 +205,7 @@ cmake --build --preset linux-cuda13 -j8 CUDA_VISIBLE_DEVICES="" ctest --preset linux-cuda13 -L '^cpu$' cmake --install build-linux-cuda13 # -> build-linux-cuda13/install cmake --install build-linux-cuda13 --component Unspecified --prefix build-linux-cuda13/install-system-cuda -(cd build-linux-cuda13 && cpack --config CPackConfig.cmake) # -> mossinfer-2.1.0-linux-cu13-x86_64.{tar.gz,zip} +(cd build-linux-cuda13 && cpack --config CPackConfig.cmake) # -> mossinfer-2.1.1-linux-cu13-x86_64.{tar.gz,zip} ``` - **glibc:** the archive needs the glibc of the machine that built it, or a newer one. Build on the oldest distribution you want to support. CI uses Ubuntu 22.04, which gives glibc 2.35. A build from a rolling distribution such as Arch will not start on Ubuntu LTS. @@ -332,10 +332,10 @@ RTX 5090, SFT, 1024², 50 steps, every model resident: The gain holds at 2048²: INT8 `high` runs 1.12 s per step against 2.77 s for bf16. 8-bit attention runs at every size; `attention` picks another kernel (`exact`, or the fp32-accumulating variants) when the caller wants one. -An `-DMOSSINFER_SM75=ON` build on an RTX 2080 8 GiB (SFT, every model resident or streamed as planned) is slower but -the same configuration ladder works: 0.80 s per 256² step for the bf16 bundle at `reference`, 0.19 s for an INT8 DiT at -`high` (6.5 GiB peak), 0.22 s for an INT4 DiT at `turbo` (3.9 GiB peak); at 1024² the INT8 `high` runs 3.8 s per step and -the VAE fp32 decode costs 10–12 s. Details in [sm75.md](sm75.md#performance). +An `-DMOSSINFER_SM75=ON` build on an RTX 2080 Ti 11 GiB (SFT) is slower but the same configuration ladder works: 0.175 s +per 256² step for the bf16 bundle at `reference`, 0.091 s for an INT8 DiT at `high` (6.3 GiB peak), 0.103 s for an INT4 DiT +at `turbo` (4.1 GiB peak); at 1024² the INT8 `high` runs 2.58 s per step and the VAE fp32 decode costs about 11 s. Details +in [sm75.md](sm75.md#performance). **Memory modes** trade host RAM for speed. `balanced` (the default) keeps the process near 8 GiB of private RAM, `min_memory` near 4 GiB (more streaming and parking, the lowest peaks), and `max_speed` uses what the machine has free (the text encoder held in RAM, pinned where there is room, and a larger embedding cache, so a new prompt starts sooner). @@ -551,7 +551,7 @@ Attach it to bug reports, together with the server's stderr log and `hello`'s `p ## Versioning - **Version numbers.** - - `mi_version()` returns the release version (`2.1.0`). + - `mi_version()` returns the release version (`2.1.1`). - `mi_abi_version()` and `MI_ABI_VERSION` give the binary interface version (1). - The server's `hello` reports its `protocol_version` (2). - **Additive changes keep ABI 1:** new functions, new fields at the end of structs, new enum values, new optional server parameters and events. diff --git a/docs/server-protocol.md b/docs/server-protocol.md index 63bfa87..e34356c 100644 --- a/docs/server-protocol.md +++ b/docs/server-protocol.md @@ -8,7 +8,7 @@ ``` moss-server # speak the protocol on stdin/stdout moss-server --parent-pid # the same, and exit when process (the launcher) exits -moss-server --version # "moss-server 2.1.0 (protocol 2, ABI 1)" +moss-server --version # "moss-server 2.1.1 (protocol 2, ABI 1)" ``` ### `--parent-pid` @@ -183,7 +183,7 @@ Creates the engine. A loaded engine is released first, because two engines would `te_host_pin` takes the pipeline's spellings (`te_host_pin_name()`); `dit_direct_io` has the server's own. -Since 2.x, `load` checks its parameters before it looks at the server's state: a malformed `load` is `invalid_params` even while a job runs (it was `busy` before). +Since 2.1.0, `load` checks its parameters before it looks at the server's state: a malformed `load` is `invalid_params` even while a job runs (it was `busy` before). ### `generate` diff --git a/docs/sm75.md b/docs/sm75.md index 5239d83..5d8e718 100644 --- a/docs/sm75.md +++ b/docs/sm75.md @@ -17,7 +17,7 @@ cmake --preset windows-msvc-cuda13-sm75 cmake --build --preset windows-msvc-cuda13-sm75 ctest --preset windows-msvc-cuda13-sm75 -L '^cpu$' cpack --config build-windows-msvc-cuda13-sm75/CPackConfig.cmake -B build-windows-msvc-cuda13-sm75 -# -> build-windows-msvc-cuda13-sm75/mossinfer-2.0.0-windows-cu13-x86_64-sm75.zip +# -> build-windows-msvc-cuda13-sm75/mossinfer-2.1.1-windows-cu13-x86_64-sm75.zip ``` Any other preset takes the option too (`-DMOSSINFER_SM75=ON`), e.g. a Linux build from a custom preset; the ordinary @@ -48,19 +48,19 @@ refused as on every non-Blackwell GPU. ## Performance -Measured on an RTX 2080 8 GiB (Turing, 46 SMs), MOSS-Image SFT, `balanced` mode, the bf16 bundle with the text encoder -streamed: +Measured on an RTX 2080 Ti 11 GiB, MOSS-Image SFT, `balanced` mode, warm runs (models resident, prompt cached), the +bf16 text encoder from the bundle: -| Configuration | Size | Denoise step | Notes | +| Configuration | Size | Denoise step | Device peak | |---|---|---|---| -| bf16 bundle, `reference` | 256² | 0.80 s | 6.9 GiB device peak | -| INT8 DiT (`moss-infer quantize … --format int8`), `high` | 256² | **0.19 s** | 6.5 GiB peak, DiT resident | -| INT4 ConvRot DiT, `turbo` | 256² | **0.22 s** | 3.9 GiB peak | -| INT8 DiT, `high` | 512² | 0.63 s | 20 steps in 22 s total | -| INT8 DiT, `high` | 1024² | 3.8 s | 8 steps in 51 s total, 6.9 GiB peak | +| bf16 bundle, `reference` | 256² | 0.175 s | 10.1 GiB | +| INT8 ConvRot DiT, `high` | 256² | **0.091 s** | 6.3 GiB | +| INT4 ConvRot DiT, `turbo` | 256² | 0.103 s | 4.1 GiB | +| INT8 ConvRot DiT, `high` | 1024² | **2.58 s** | 7.5 GiB | +| INT4 ConvRot DiT, `turbo` | 1024² | 2.59 s | 5.3 GiB | -The VAE decodes in fp32 on sm_75: about 0.4 s at 256², 1.6 s at 512² and 10–12 s at 1024². The bf16 text encoder is -roughly 3–5 s per prompt when streamed (its ~16 GB do not fit 8 GiB of VRAM); it is planned like on any other device. +The VAE decodes in fp32 on sm_75: about 0.5 s at 256² and 11 s at 1024². An 8 GiB card (RTX 2080) also runs all three +tiers, with the bf16 text encoder streamed. ## Numerics @@ -79,16 +79,17 @@ roughly 3–5 s per prompt when streamed (its ~16 GB do not fit 8 GiB of VRAM); ## Verification -- `tools/sm75_check.cu` (not installed, built with `MOSSINFER_SM75=ON`) runs the Turing kernels against host - references: dtype conversion, exact attention at both head dimensions and engine-shaped varlen/GQA cases (also +- `tools/sm75_check.cu` (not installed, built with `MOSSINFER_SM75=ON`; every case passes on an RTX 2080 Ti) runs the + Turing kernels against host references: dtype conversion, exact attention at both head dimensions and engine-shaped varlen/GQA cases (also causal with key lengths unlike the query lengths, and that q/k/v are left untouched), the fp16 GEMM (bf16 and f32 out, a row above fp16's range, several row chunks), the INT8 GEMM (bit-exact), and the VAE convolution and attention. - `tests/unit/test_sm75.cpp` covers the build's presence and, on a Turing device, the weight conversion round trip and - the GEMM rows' per-row scale. + the GEMM rows' per-row scale. Both pass on an RTX 2080 Ti. - The refusals are exercised on the RTX 2080: `moss-infer generate --init-image …` and `--lora …` end with `not_supported` (CLI exit status 1, nothing written), and `moss-server` reports the same code in the `done` event. - End to end on an RTX 2080: `moss-infer generate` and `moss-server` (protocol 2) produce the same PNG bytes for the - `reference`, `high` and `turbo` tiers; the report's `quality.gpu_arch` is `sm_75`. + `reference`, `high` and `turbo` tiers; the report's `quality.gpu_arch` is `sm_75`. On an RTX 2080 Ti, separate + `moss-server` processes produce the same PNG bytes at all three tiers, at 256² and (`high`, `turbo`) 1024². - A build with `MOSSINFER_SM75=OFF` compiles and links unchanged and keeps the same CPU test baseline. ## Not supported on sm_75 yet @@ -108,4 +109,4 @@ roughly 3–5 s per prompt when streamed (its ~16 GB do not fit 8 GiB of VRAM); - Forced INT8 GEMM tiles (`Int8Tile`, a test and benchmark option): sm_75 runs one tile (64x128x64), so a forced tile is refused rather than ignored. -The VAE decode is fp32 SIMT on sm_75, so a 1024² decode costs about 10–12 s; a future fp16 path would cut that. +The VAE decode is fp32 SIMT on sm_75, so a 1024² decode costs about 11 s; a future fp16 path would cut that. diff --git a/include/mossinfer/mossinfer.h b/include/mossinfer/mossinfer.h index abecc2a..2fba555 100644 --- a/include/mossinfer/mossinfer.h +++ b/include/mossinfer/mossinfer.h @@ -75,7 +75,7 @@ extern "C" { #define MI_VERSION_MAJOR 2 #define MI_VERSION_MINOR 1 -#define MI_VERSION_PATCH 0 +#define MI_VERSION_PATCH 1 /* Incremented on every incompatible change of the binary interface (see the conventions above). */ #define MI_ABI_VERSION 1