[bettercallcaleb]

Problem

On a Qwen3.8-27B IQ2_XS dense GGUF, CUDA prefill fails with:

Error: gemm inside batch: x must be quantized already (SAME_X)

The DeltaNet gate GEMM uses an integer tensor-core path and sets SAME_X inside a parallel batch. The earlier GEMM need not have populated the matching quantized activation cache, so the batch cannot safely quantize it.

Fix

Prepare the gate's quantized input before starting the batched DeltaNet scan and gate GEMM. Implement this on CUDA and Vulkan, preserving the existing parallel scheduling and the float-weight fast path.

Verification

• Reproduced the original failure on an RTX 3060 with a Qwen3.8-27B IQ2_XS GGUF.
• After the fix, a short bench --pp 256 --tg 32 --reps 1 --mtp 0 run completed on both CUDA and Vulkan. CUDA: 296.5 prefill / 23.9 decode tok/s; Vulkan: 64.3 prefill / 16.9 decode tok/s. These are single-run diagnostics, not comparative performance claims.
• Added a synthetic Q4_K batched GEMM regression test comparing against the non-batched result on both backends.
• cargo test --locked -- --skip spirv_is_fresh passed. The unfiltered suite encounters an existing local GLSL compiler error in the shader regeneration test; no shaders changed.
• git diff --check passed.

[Vasyl-Trefilov]

Thanks for tracking this down. I don't have that GGUF, but my guess is that its qkv (ssm_in) is IQ2_XS. IQ2_XS takes the f16 GEMM (it is off the int8 list, #31), so nothing has quantized h by the time the gate GEMM asks for SAME_X inside the batch. It merges cleanly onto current main and the suite passes here.

Three requests before merging:

1. The beta/alpha batch has the same hazard. prefill.rs batches beta and alpha with SAME_X a few lines earlier, right after the same qkv GEMM. On your model they are probably float, so they passed, but a GGUF with IQ2_XS ssm_in and quantized beta/alpha would fail there the same way. Could you move the prepare step to just after the qkv GEMM, so it covers all three? They all read h with ne columns, so one quantized copy serves them. For that to work, prequant_gemm_x should return early when the cache already holds that key, so a second call does not quantize again.

2. One copy of the quantize step per backend. prequant_gemm_x repeats what gemm_raw does (buffer size, alloc, quant_q8, key). Could you factor that into a small helper that both call? Then the size formula and the key are defined in one place, and they cannot drift apart.

3. Turing. If cuda: add a Turing (sm_75) path #6 lands first, the CUDA side also needs !self.int8_mma || in the early return, the same check as in gemm_raw. Otherwise Turing quantizes a buffer that nothing reads. I can add that when merging the second of the two.

Small point: the batched and non-batched results come from the same quant_q8 and the same kernel, so the test could compare bits (to_bits()) instead of using a 1e-4 tolerance. That would also catch a future change that makes them differ slightly.