CUDA_ERROR_ILLEGAL_ADDRESS on first forward pass — RTX 5090 (sm_120), CUDA 13, Qwen3.8-Flash-Next (Vulkan works) #34

Summary

On an RTX 5090 (Blackwell, sm_120) with CUDA 13, every CUDA run of Qwen3.8-Flash-Next IQ3_XXS fails on the first forward pass with CUDA_ERROR_ILLEGAL_ADDRESS. Zero tokens are generated. The same binary and model work on the Vulkan backend on the same GPU.

Environment

• RustingStrata v1.2.2 (commit 757fd0c), built from source with cargo build --release
• rustc 1.100.0
• Ubuntu 26.04.1 LTS, kernel 7.0.0-43-generic
• NVIDIA GeForce RTX 5090 32 GB, driver 595.91.07, CUDA 13.0 (NVRTC libnvrtc.so.13)
• 32 cores, 182 GB RAM
• An AMD iGPU is also present (Vulkan device 0); the 5090 is Vulkan device 1
• Model: Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS (2 split files, 44 GB, arch qwen4exp). It does not fit in VRAM, so this is the hybrid CPU+GPU experts path.

Steps to reproduce

rustingstrata serve --model Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf --gpu --ctx 4096 --mtp 0 --addr 127.0.0.1:8180

Then send any /v1/chat/completions request (a 20-token prompt is enough).

calibrate --model <same gguf> fails the same way: the prefill chunk sweep finishes (chunk 1024: 1486 tok/s, 2048: 2202, 4096: 2942), and the next step errors with the same CUDA_ERROR_ILLEGAL_ADDRESS. No calibration file is written.

Log (serve, --chunk 1024 --vram-margin 3072)

cuda device 0: NVIDIA GeForce RTX 5090, sm_120
cuda shared memory: opt-in 101376 B per block, attn_decode_gqa*: 2112 B static, dynamic max 73728 B, carveout default
cuda shared memory: opt-in 101376 B per block, attn_prefill_*: 0 B static, dynamic max 72704 B, carveout default
cuda shared memory: opt-in 101376 B per block, gemvn_*: 0 B static, dynamic max 101376 B, carveout default
plan: weights+state+scratch 4320 MB (KV 96 MB for 4096 tokens, prefill 438 MB), expert slots 23390 MB, resident experts 14046/24576
expert traffic: 22.2 MB per layer active set (top-10), 799.3 MB per token over 48 MoE layers
chat template: Xml
loaded Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002
request: POST /v1/chat/completions ... prompt=20 tok stream=true tools=0
done: ... prompt=20 tok (0 cached) -> 0 tok in 0.02 s, decode 0.0 tok/s, prefill 666666666.7 tok/s, finish=error
request 1: DriverError(CUDA_ERROR_ILLEGAL_ADDRESS, "an illegal memory access was encountered")
stopped
expert hit rate 0.0% (0 misses), cache swaps 0
worker busy 0.0 µs/token, worker stalls 0
mailbox traffic: 48 GPU writes, 0 host routing reads (fast path off, 14046/24576 experts resident)

RUST_BACKTRACE=1 printed no backtrace, only the error above.

What I tried

All of these crash the same way on the first request:

• default flags (15.0K of 24.6K experts resident, about 31.6 GB VRAM)
• --vram-margin 3072 (13448 experts resident)
• --vram-margin 6144 (11605 experts resident)
• --chunk 1024 --vram-margin 3072 (14046 experts resident)

So it does not look like VRAM pressure.

Vulkan works on the same card

--backend vulkan --device 1 loads and generates correctly (greedy output matches llama.cpp in the first ~400 characters on a 3K-token prompt). Rough numbers on this machine, against llama.cpp llama-server on the same model:

• prefill, 3001-token prompt: about 1570–1680 tok/s (llama.cpp about 580)
• decode after the 3K prompt: about 66–75 tok/s (llama.cpp about 63–66)

Side note: without --device, Vulkan picks the AMD iGPU (device 0) and fails with "Not enough memory for command submission".

I'm happy to run a debug build or test a patch on this machine.