Repository navigation
Conversation
…a, A100 40 GB results; GTX 1080 Ti does not run Tested rows and calibrated benchmarks for eight cards on a second cluster, the kernel tests that need STRATA_VK_F16_TRUNC=1 on Turing as well as Volta, and what happens on a GTX 1080 Ti (neither backend runs). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014eNPMupWL2tifApEBfcCgi
gemm_f's h2 packed with packFloat2x16(f16vec2(..)), which the NVIDIA Volta and Turing Vulkan drivers truncate toward zero. Build the pairs from rh (software round-to-nearest) and packHalf2x16, so the f16 GEMM sees the same operands as CUDA's __float2half on every device. This removes the STRATA_VK_F16_TRUNC test escape hatch and makes the cross-backend GEMM check strict again. It also merges the duplicate V100 rows and fixes the 2080 Ti resident-expert range from RustingStudio#27.
gemm_f's h2 packed with packFloat2x16(f16vec2(..)), which the NVIDIA Volta and Turing Vulkan drivers truncate toward zero. Build the pairs from rh (software round-to-nearest) and packHalf2x16, so the f16 GEMM sees the same operands as CUDA's __float2half on every device. This removes the STRATA_VK_F16_TRUNC test escape hatch and makes the cross-backend GEMM check strict again. It also merges the duplicate V100 rows and fixes the 2080 Ti resident-expert range from RustingStudio#27.
525a976 to
ea4c8a8
Compare
|
Thanks, this is a really useful report. I pushed a fix to your branch: gemm_f now rounds f16 GEMM inputs in software (rh + packHalf2x16), so the Volta/Turing truncation is gone — STRATA_VK_F16_TRUNC and the test workaround are deleted. Also merged the duplicate V100 rows and fixed the 2080 Ti resident range. CI passes and the Vulkan kernel suite passes on an RTX 3060.
|
|
Thanks for the fix. Re-ran on 1. Test suite: all four cards pass without the variable.
2. Vulkan prefill is 2-6% slower with the software conversion. Same bench as in the doc (Q2_K_XL,
The pp 4096 ranges do not overlap on any of the three, and CUDA on the same nodes is unchanged, so it is the shader and not the node. The calibrated MTP run on the V100 shows the same -2.9% at pp 4096 (1597.7 -> 1551.9). Decode is not affected. The V100 32 GB bench is still queued. Whether 2-5% of Vulkan prefill on Turing/Volta is worth a specialization constant is your call; I can re-measure if you add one. 3. sm_61: PR coming separately (early "requires sm_70" error and the 4. Bigger models: fair point. Q8_0 (37.8 GB) and BF16 (71.1 GB), the same files as in |
…cards RTX 2080 Ti, Quadro RTX 6000 and both V100s pass 185 tests at ea4c8a8 with the variable unset. Vulkan prefill at pp 4096 is 2-5% lower than at f47e370 on the same nodes. Q8_0 bench on A100 40 GB, A40, RTX 4500 Ada, Quadro RTX 6000, RTX 2080 Ti and V100 16 GB. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014eNPMupWL2tifApEBfcCgi
|
Pushed One correction to my comment above: the Quadro RTX 6000 re-run landed on another card of the same node, and CUDA pp 4096 moved +1.2% there, so its -1.8% on Vulkan is not evidence by itself. The RTX 2080 Ti (-5.0%) and V100 16 GB (-2.8%) hold: CUDA on those nodes moved by 0.0% and +0.1%. Q8_0 in short (CUDA, pp 4096,
Still queued: Q8_0 on the RTX A6000 and V100 32 GB, the V100 32 GB bench re-run, and BF16 on the five larger cards (the first shard is still downloading). The sm_61 change is #28. |
|
@yugt I also worried about DeepSeekV4.1 1017 performance and it's popular big local MoE model so could you rerun on the same nodes, page cache warm, with as many cores as the scheduler gives (ideally 32) and a budget sweep near the card limit? A100 80 GB: STRATA_DS_GPU=1 STRATA_DS_PROF=1 STRATA_DS_GPU_EXPERTS=60,64,68,70 Same prompt and -n 128. Please report, per budget: end-to-end tok/s, VRAM hit rate, expert/attention ms, and read_bytes so we know the cache is warm. If you can, add a llama.cpp baseline on the same node/file/prompt (--cpu-moe, current build) so we know what the target is. |
|
@yugt I don't know what you do and why you have so much GPUs and etc. but what do you think if I will try to bring RustingStrata closer to vLLM, so people like you can use it to sell API and maybe it will be even better then vLLM, I just want to know if you need this and if yes, then I will try to make it |
Eight more NVIDIA cards from a second Slurm cluster, each benchmarked before and after
calibrateon CUDA and Vulkan, plus the test suite, and one card that does not run. Docs only.Runs at
f47e370,Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf, one GPU per job with 8 cores and 48 GB RAM,cargo testwith--test-threads=1.pp 4096 prefill / decode, tok/s ("MTP off" is the calibrated run with
--mtp 0; "calibrated" uses the MTP thatcalibratechose):cargo testThe cards sit in hosts with different CPUs, so compare within a row. The model fits in VRAM everywhere except the 2080 Ti (8213-8712 experts resident, hit rate 99.7-99.9%).
Two things I ran into
1. Four Vulkan kernel tests fail on Turing too, not only on the V100. On the RTX 2080 Ti, the Quadro RTX 6000 and both V100s,
f16_weight_uploads_as_f32,gemm_matches_reference_for_every_type,gemv_partial_superblocks_and_q2_0andvulkan_type_matrixfail with relative errors around 5e-4. WithSTRATA_VK_F16_TRUNC=1all four cards pass 185 of 185. The two Turing cards report cooperative matrices (f16 and int8) and run driver 615.71.09, the same driver on which the Ampere and Ada cards pass without the variable. So the truncating f16 conversion is not specific to the Volta driver; the comment attests/kernels.rs:339says it is. The Turing and V100 Vulkan rows say "Runs, test suite passes withSTRATA_VK_F16_TRUNC=1" and note that the variable only changes what the tests expect.2. A GTX 1080 Ti (Pascal, driver 580.95.05) runs on neither backend.
--backend vulkan:no usable Vulkan device 0 (0 usable; NVIDIA GeForce GTX 1080 Ti: missing shaderFloat16).--backend cudawith NVRTC 12.6: the startup line iscuda device 0: NVIDIA GeForce GTX 1080 Ti, sm_61 (Volta, m8n8k4), thenNVRTC compile of kernels.cu failedwithdefault_program(2321): error: identifier "__nanosleep" is undefined.The README said "Pascal and older use Vulkan" and
SUPPORTED_GPUS.mdsaid "Use--backend vulkan; not tested"; both now say what happens on this card, and the README's Vulkan requirements listshaderFloat16.cargo teston that card reports 185 passed only because every GPU test skipped (no usable Vulkan device, and that job had no NVRTC loaded), so I did not count it as tested.Would you like a small separate PR for this: an early "sm_61 is not supported" error for devices below sm_70 instead of the NVRTC compile failure and the "Volta" label? I have not written it.
What is not in here
STRATA_VK_F16_TRUNC=1runs got the variable throughsbatch --export=ALL; the logs do not echo it. Three of the four reruns landed on another node of the same card type than the run without it (the Quadro RTX 6000 was the same node).vkDestroyDevicehang from A100, A30, L40S and H100: tested rows, results, and default-model test checks by file name #24.Full tables (mean, min-max of 3 repetitions), what
calibratechose, hosts and startup lines are indocs/more-nvidia-results.md.🤖 Generated with Claude Code
https://claude.ai/code/session_014eNPMupWL2tifApEBfcCgi