moss-infer-vulkan: the Vulkan fallback program, with Linux and Windows packages

.github/workflows/build-vulkan.yml

@@ -0,0 +1,143 @@
+# Reusable build of moss-infer-vulkan, the standalone Vulkan program (vulkan/), called by ci.yml and release.yml. Per
+# platform: the pinned Vulkan SDK (headers and glslc, checked against LunarG's SHA-256), configure and build vulkan/,
+# the CPack archive. On Linux also the backend's checks, run on lavapipe (Mesa's CPU Vulkan driver: the plain kernels)
+# with the Khronos validation layer from the SDK, since hosted runners have no GPU. No CUDA is involved.
+name: build-vulkan
+
+on:
+  workflow_call:
+    inputs:
+      retention-days:
+        description: Days to keep the uploaded packages
+        type: number
+        default: 7
+
+permissions:
+  contents: read
+
+env:
+  VULKAN_SDK_VERSION: 1.4.357.0
+
+jobs:
+  vulkan:
+    name: ${{ matrix.platform }}-vulkan
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 120
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - os: ubuntu-22.04
+            platform: linux
+            sdk_sha256: 0f09bf6a0625e346bf004be70b92907e934a4c76606b323441b2baf3a5a0e66d
+          - os: windows-2022
+            platform: windows
+            sdk_sha256: 81f474711e9042f4cd22b31b2f7a8870db2e428b21586fb43dd80150be97310d
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      - uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
+
+      - uses: ilammy/msvc-dev-cmd@0b201ec74fa43914dc39ae48a89fd1d8cb592756 # v1.13.0
+        if: matrix.platform == 'windows'
+
+      - name: Toolchain and lavapipe (Linux)
+        if: matrix.platform == 'linux'
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends g++-12 zlib1g-dev mesa-vulkan-drivers libvulkan1
+          sudo apt-get clean
+          echo "CXX=g++-12" >> "$GITHUB_ENV"
+
+      # The SDK (330 MB on Linux, 290 MB on Windows) is cached per version: LunarG's server can be slow from the runners.
+      - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        id: sdk-cache
+        with:
+          path: ${{ runner.temp }}/vulkan-sdk
+          key: vulkan-sdk-${{ matrix.os }}-${{ env.VULKAN_SDK_VERSION }}-v1
+
+      - name: Vulkan SDK download (Linux)
+        if: matrix.platform == 'linux' && steps.sdk-cache.outputs.cache-hit != 'true'
+        run: |
+          f="vulkansdk-linux-x86_64-$VULKAN_SDK_VERSION.tar.xz"
+          curl -fsSL --retry 5 --retry-all-errors --retry-delay 15 --connect-timeout 30 -o "$RUNNER_TEMP/$f" \
+            "https://sdk.lunarg.com/sdk/download/$VULKAN_SDK_VERSION/linux/$f"
+          echo "${{ matrix.sdk_sha256 }}  $RUNNER_TEMP/$f" | sha256sum -c -
+          mkdir -p "$RUNNER_TEMP/vulkan-sdk"
+          tar -xJf "$RUNNER_TEMP/$f" -C "$RUNNER_TEMP/vulkan-sdk"
+          rm "$RUNNER_TEMP/$f"
+
+      - name: Vulkan SDK download (Windows)
+        if: matrix.platform == 'windows' && steps.sdk-cache.outputs.cache-hit != 'true'
+        shell: pwsh
+        run: |
+          $f = "vulkansdk-windows-X64-$env:VULKAN_SDK_VERSION.exe"
+          $exe = Join-Path $env:RUNNER_TEMP $f
+          curl.exe -fsSL --retry 5 --retry-all-errors --retry-delay 15 --connect-timeout 30 -o $exe `
+            "https://sdk.lunarg.com/sdk/download/$env:VULKAN_SDK_VERSION/windows/$f"
+          if ($LASTEXITCODE -ne 0) { throw "download failed" }
+          $hash = (Get-FileHash -Algorithm SHA256 $exe).Hash.ToLower()
+          if ($hash -ne "${{ matrix.sdk_sha256 }}") { throw "SHA-256 mismatch: $hash" }
+          $root = Join-Path $env:RUNNER_TEMP "vulkan-sdk"
+          Start-Process -FilePath $exe -Wait -NoNewWindow -ArgumentList @("--root", $root, "--accept-licenses", "--default-answer", "--confirm-command", "install")
+          Remove-Item $exe
+
+      - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        if: steps.sdk-cache.outputs.cache-hit != 'true'
+        with:
+          path: ${{ runner.temp }}/vulkan-sdk
+          key: ${{ steps.sdk-cache.outputs.cache-primary-key }}
+
+      - name: Vulkan SDK environment
+        shell: bash
+        run: |
+          if [ "${{ matrix.platform }}" = linux ]; then
+            sdk="$RUNNER_TEMP/vulkan-sdk/$VULKAN_SDK_VERSION/x86_64"; bin="$sdk/bin"
+          else
+            sdk="$RUNNER_TEMP/vulkan-sdk"; bin="$sdk/Bin"
+          fi
+          echo "VULKAN_SDK=$sdk" >> "$GITHUB_ENV"
+          echo "$bin" >> "$GITHUB_PATH"
+          "$bin/glslc" --version
+
+      - name: Configure
+        shell: bash
+        run: |
+          tools=OFF
+          extra=""
+          if [ "${{ matrix.platform }}" = linux ]; then tools=ON; extra="-DMOSSINFER_STATIC_LIBSTDCXX=ON"; fi
+          cmake -S vulkan -B build-vulkan -G Ninja -DCMAKE_BUILD_TYPE=Release -DMOSSINFER_VULKAN_TOOLS=$tools $extra
+
+      - name: Build
+        run: cmake --build build-vulkan
+
+      - name: Checks on lavapipe (Linux)
+        if: matrix.platform == 'linux'
+        run: |
+          export VK_DRIVER_FILES="$(ls /usr/share/vulkan/icd.d/lvp_icd.*.json | head -n 1)"
+          export VK_LAYER_PATH="$VULKAN_SDK/share/vulkan/explicit_layer.d"
+          export LD_LIBRARY_PATH="$VULKAN_SDK/lib"
+          export VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation
+          build-vulkan/moss-infer-vulkan --list-devices
+          # Every check runs, so one run reports all failures.
+          status=0
+          build-vulkan/vulkan_block_check --no-block || status=1
+          build-vulkan/vulkan_te_kernel_check || status=1
+          build-vulkan/vulkan_vae_check --kat-only || status=1
+          exit $status
+
+      - name: Package
+        shell: bash
+        run: |
+          cd build-vulkan
+          cpack
+          ls -l mossinfer-*-vulkan-*
+
+      - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
+        with:
+          name: mossinfer-vulkan-${{ matrix.platform }}
+          path: |
+            build-vulkan/mossinfer-*-vulkan-*.zip
+            build-vulkan/mossinfer-*-vulkan-*.tar.gz
+          retention-days: ${{ inputs.retention-days }}
+          if-no-files-found: error

.github/workflows/ci.yml

@@ -28,3 +28,7 @@ jobs:
     with:
       retention-days: 7
       sccache-recache: ${{ inputs.recache == true }}
+  vulkan:
+    uses: ./.github/workflows/build-vulkan.yml
+    with:
+      retention-days: 7

.github/workflows/release.yml

@@ -1,7 +1,8 @@
 # Tag vX.Y.Z (a release) or vX.Y.Z-rcN, N >= 1 (a release candidate, published as a pre-release that does not become
 # "latest"): build and check all four standard flavours, the Windows CUDA 13 SM75 flavour and the Linux CUDA 12 SM70
-# flavour (build.yml), then publish
-# their archives (the full package and the -system-cuda variant of each) and a SHA256SUMS file as a GitHub release.
+# flavour (build.yml) and moss-infer-vulkan for Linux and Windows (build-vulkan.yml), then publish
+# their archives (the full package and the -system-cuda variant of each CUDA flavour, one archive per Vulkan platform)
+# and a SHA256SUMS file as a GitHub release.
 # X.Y.Z must equal the MI_VERSION_* of
 # include/mossinfer/mossinfer.h and an rc must come before vX.Y.Z (.github/scripts/release_version.py), the tagged
 # commit must be on main with a successful ci run (.github/scripts/release_gate.sh), and the release must not exist
@@ -92,9 +93,15 @@ jobs:
     with:
       retention-days: 30
 
+  vulkan:
+    needs: version
+    uses: ./.github/workflows/build-vulkan.yml
+    with:
+      retention-days: 30
+
   # Checks and gathers the release assets with a read-only token; on a dispatch its artifact is the dry run's result.
   package:
-    needs: [version, build]
+    needs: [version, build, vulkan]
     runs-on: ubuntu-22.04
     env:
       ARCHIVE_VERSION: ${{ needs.version.outputs.archive_version }}
@@ -112,7 +119,7 @@ jobs:
           for full in mossinfer-*.zip mossinfer-*.tar.gz; do
             [ -f "$full" ] || continue
             case "$full" in "mossinfer-$ARCHIVE_VERSION-"*) ;; *) echo "$full is not version $ARCHIVE_VERSION"; exit 1 ;; esac
-            case "$full" in *-system-cuda.*) continue ;; esac
+            case "$full" in *-system-cuda.* | *-vulkan-*) continue ;; esac
             case "$full" in *.tar.gz) ext="tar.gz"; base="${full%.tar.gz}" ;; *) ext="zip"; base="${full%.zip}" ;; esac
             test -f "$base-system-cuda.$ext" || { echo "missing $base-system-cuda.$ext"; exit 1; }
             n=$((n + 1))
@@ -125,6 +132,9 @@ jobs:
           sm70="mossinfer-$ARCHIVE_VERSION-linux-cu12-x86_64-sm70.zip"
           test -f "$sm70" || { echo "missing SM70 package $sm70"; exit 1; }
           test -f "${sm70%.zip}-system-cuda.zip" || { echo "missing SM70 system-CUDA package"; exit 1; }
+          for vk in "mossinfer-$ARCHIVE_VERSION-linux-vulkan-x86_64.tar.gz" "mossinfer-$ARCHIVE_VERSION-windows-vulkan-x86_64.zip"; do
+            test -f "$vk" || { echo "missing Vulkan package $vk"; exit 1; }
+          done
       - name: SHA256SUMS
         shell: bash
         working-directory: dist

README.md

@@ -4,7 +4,7 @@ A native C++/CUDA inference engine for the MOSS-Image text-to-image model (5B Di
 
 Models: MOSS-Image Base and SFT, in the official folders and common community formats; other models are refused ([supported models](#supported-models)).
 
-Version 2.1.2. Supported: Linux x86-64 and Windows x64 (MSVC), each with a CUDA 12 and a CUDA 13 build, as a full package and a `-system-cuda` package. Turing (compute capability 7.5, e.g. RTX 2080) is supported by builds configured with `-DMOSSINFER_SM75=ON` (the `windows-msvc-cuda13-sm75` preset packages it): fp16 tensor-core GEMMs replace the bf16 ones, INT8 and INT4 ConvRot files run the INT8 GEMMs, the exact attention and the VAE run sm_75 kernels, all three quality tiers work, and an unset attention resolves to `exact`. See [docs/sm75.md](docs/sm75.md) for what runs, the measured speed on an RTX 2080 Ti and the remaining gaps. Volta (Tesla V100, sm_70) is supported by the CUDA 12 `linux-cuda12-sm70` preset; see [docs/sm70.md](docs/sm70.md) for its kernels, validation and performance. The full support matrix is in [docs/integration.md](docs/integration.md#support-status).
+Version 2.1.2. Supported: Linux x86-64 and Windows x64 (MSVC), each with a CUDA 12 and a CUDA 13 build, as a full package and a `-system-cuda` package. Turing (compute capability 7.5, e.g. RTX 2080) is supported by builds configured with `-DMOSSINFER_SM75=ON` (the `windows-msvc-cuda13-sm75` preset packages it): fp16 tensor-core GEMMs replace the bf16 ones, INT8 and INT4 ConvRot files run the INT8 GEMMs, the exact attention and the VAE run sm_75 kernels, all three quality tiers work, and an unset attention resolves to `exact`. See [docs/sm75.md](docs/sm75.md) for what runs, the measured speed on an RTX 2080 Ti and the remaining gaps. Volta (Tesla V100, sm_70) is supported by the CUDA 12 `linux-cuda12-sm70` preset; see [docs/sm70.md](docs/sm70.md) for its kernels, validation and performance. Other GPUs (AMD, Intel, older NVIDIA) can run txt2img at the reference tier through `moss-infer-vulkan`, a separate and slower Vulkan program in its own package ([docs/vulkan.md](docs/vulkan.md)). The full support matrix is in [docs/integration.md](docs/integration.md#support-status).
 
 The caller picks the quality tier: `reference` (bf16, exact), `high` (the loaded DiT's GEMMs with 8-bit attention) or `turbo` (a 4-bit DiT). An INT8 DiT with `high` is the everyday choice: 2.5 times the bf16 speed on an RTX 5090, in 40 % less VRAM ([performance](docs/integration.md#performance)). Upgrading from 1.x: [migration](docs/integration.md#migrating-from-1x).
 
@@ -30,6 +30,7 @@ Lowest VRAM against normal use at 1024×1024. `min_memory` streams the DiT and c
 - [docs/integration.md](docs/integration.md): building the engine into an application. Start here.
 - [docs/sm70.md](docs/sm70.md): Volta (Tesla V100) support: CUDA 12 build, exact INT8 accumulation, validation and performance.
 - [docs/sm75.md](docs/sm75.md): Turing (compute capability 7.5) support: build and package, what runs, measured speed on an RTX 2080 Ti, limitations.
+- [docs/vulkan.md](docs/vulkan.md): `moss-infer-vulkan`, the Vulkan fallback for GPUs the CUDA engine does not support.
 - [docs/c-api.md](docs/c-api.md): the C library, `include/mossinfer/mossinfer.h`.
 - [docs/server-protocol.md](docs/server-protocol.md): `moss-server`, the JSON-lines sidecar process.
 

docs/vulkan.md

@@ -0,0 +1,85 @@
+# Vulkan backend
+
+`moss-infer-vulkan` runs MOSS-Image on Vulkan. It is the fallback for GPUs the CUDA engine does not support (AMD, Intel,
+older NVIDIA cards): a standalone program in its own package (`mossinfer-VERSION-linux-vulkan-x86_64.tar.gz`,
+`mossinfer-VERSION-windows-vulkan-x86_64.zip`) that needs a Vulkan driver and neither CUDA nor an NVIDIA driver. It is
+slower than CUDA; on a supported NVIDIA GPU, use `moss-infer`.
+
+It runs txt2img with the official Base and SFT bf16 files, the official euler sampler and schedule and classifier-free
+guidance with the uncond skip layers. The tier is `reference`. Everything else is refused with a message that names
+the option: other samplers and schedulers, LoRAs, img2img and inpainting, hires fix, control adapters, references,
+the lossy accelerators, quantized or GGUF files, and the CUDA memory options. The tokenizer, the seed's initial noise
+and the schedule are the engine's own, so a seed gives the same inputs as on CUDA.
+
+```sh
+moss-infer-vulkan --model MOSS-Image/base --prompt "a red apple on a wooden table" --size 1024x1024 -o apple.png
+moss-infer-vulkan --list-devices
+```
+
+The flags are those of `moss-infer generate` that the backend runs, with the same defaults (`--negative`, `--size`,
+`--width`, `--height`, `--steps`, `--cfg`, `--shift`, `--seed`, `--skip-layers`, `--max-seq`, `--latent-out`,
+`--no-metadata`, `--quiet`; `--tier` accepts `reference`). `--device N` picks a device (default: the first usable one, a
+discrete GPU first) and `--kernels coop|coop-f16|plain` the kernels (default: the first in that order the device has);
+both are printed at the start of every run. Progress goes to stderr one line per event: `text encoder`, `denoise <step>/<steps>
+<seconds>`, `vae`, then a `done:` summary. The exit status is 0 on success, 2 for a usage error or an unsupported
+request, 1 for any other failure.
+
+## Kernels
+
+| Kernels | Needs | Used for |
+|---|---|---|
+| `coop` | Vulkan 1.3, `VK_KHR_cooperative_matrix` and `VK_KHR_shader_bfloat16` with 16x16x16 bf16 fragments, a compute subgroup size fixable at 32, 40 KiB of shared memory | GEMMs, attention and the VAE's convolutions on matrix cores |
+| `coop-f16` | the same with 16x16x16 f16 fragments instead of bf16 ones | devices with matrix cores but no bf16 fragments (Mesa's RADV exposes them only on RDNA4, for one): the GEMMs and attention on matrix cores, the VAE as `plain` |
+| `plain` | Vulkan 1.2, buffer device addresses, 64-bit integers, 32 KiB of shared memory | every other device; no cooperative matrices, 16-bit arithmetic or particular subgroup size |
+
+`coop-f16` computes what `coop` computes. The weights and activations stay bf16; each GEMM operand row is scaled by a
+power of two that brings it into f16's range as it is loaded, where a bf16 value is an f16 value, and the result is
+scaled back, so the products are the same. Only attention's q, k and v convert unscaled: their rare values below 2^-14
+lose bits, and a value above 65504 would become infinite (the model's stay far below). On the golden case's text encoder and on real DiT blocks it gives the same figures as `coop`, and the
+same step 0 accuracy below; whole images then differ as any two backends' do (see below).
+
+## Memory
+
+The text encoder streams its 32 layers from the model files, one layer on the device at a time. The DiT keeps as many
+of its 42 blocks on the device as fit next to the run's activations, from the driver's memory budget
+(`VK_EXT_memory_budget`). The other blocks stay in host memory and are copied on the transfer queue while the block
+before them runs. So a GPU with less than the DiT's 9.5 GiB still runs it, at the cost of host memory and bus traffic.
+The VAE decodes the whole image at once: about 1.6 GiB at 1024x1024, 3.1 GiB at 1456x1456 and 6 GiB at 2048x2048. A
+decode the device cannot hold is refused before the text encoder runs. The text encoder and the DiT are on the device
+one after the other, never together.
+
+Long operations are split into submits of bounded work, so no submit runs long enough to trip the Windows display
+driver's watchdog on a slow GPU.
+
+## Accuracy
+
+Against the official PyTorch pipeline's golden data at 1024x1024 (`tests/golden/g_fridge_1024`), on the `coop` path:
+
+| Stage | Result |
+|---|---|
+| Text encoder, last hidden state | cosine 0.99988 (prompt), 0.99980 (negative prompt); torch itself with one kernel swapped: 0.99989, 0.99983 |
+| DiT, step 0 prediction | cosine 0.999976 (cond), 0.999983 (uncond); the CUDA engine's test asks for 0.99988 |
+| VAE decode of the golden latent | PSNR 69.4 dB against the golden PNG, 99.25 % identical bytes, max difference 1 |
+
+Whole images keep the composition and the text but not the pixels: 50 sampling steps magnify any difference. The
+golden 1024x1024 case through `moss-infer` ends with a final latent at cosine 0.974 to the official pipeline's (the CUDA
+backend: 0.992, being nearly bit-identical to torch op by op) and an image at 30.8 dB PSNR. That is the sampler's
+sensitivity, not an error of the backend: the same Vulkan steps on torch's own text embeddings instead of the Vulkan
+encoder's (cosine 0.9998 apart, about torch's own variation when one of its kernels changes) end 0.971 apart from each
+other. A 512x512, 20-step image with seed 7 is 32.9 dB PSNR from the CUDA engine's image of the same command: the same apple,
+details redrawn.
+
+## Building
+
+`vulkan/` is its own CMake project and needs no CUDA: Vulkan headers that define `VK_KHR_shader_bfloat16` and a `glslc`
+that compiles `GL_EXT_bfloat16`, as in the Vulkan SDK 1.4.357 the release builds use. The program opens the Vulkan
+loader at run time, so it links nothing of Vulkan.
+
+```sh
+cmake -S vulkan -B build-vulkan -G Ninja -DMOSSINFER_VULKAN_TOOLS=OFF
+cmake --build build-vulkan
+cpack --config build-vulkan/CPackConfig.cmake -B build-vulkan
+```
+
+With the tools on (the default) it also builds the backend's checks and benchmarks; see
+[vulkan/README.md](../vulkan/README.md).