Skip to content

feat: EAGLE3 speculative decoding, hybrid N-gram cascade, and RDNA3.5 Vulkan optimizations - #36

Merged
Vasyl-Trefilov merged 1 commit into
RustingStudio:mainfrom
ybenyedder:upstream/eagle3-ngram-vulkan
Nov 15, 2026
Merged

Vasyl-Trefilov merged 1 commit into
RustingStudio:mainfrom
ybenyedder:upstream/eagle3-ngram-vulkan

Conversation

@ybenyedder

Copy link
Copy Markdown

Summary

  • EAGLE3 standalone draft support (--mtp-model): loads EAGLE3 GGUF or auto-converts .safetensors + config.json (including angelslim, tengyunw, and RedHatAI/speculators transformer_layer_config / layers.0.* aliases) in pure Rust (src/gguf.rs, src/model.rs), with Vulkan graph capture, d2t remap, and non-throttled adaptive MTP depth (src/gpu_model.rs).
  • Hybrid N-gram + EAGLE3 speculative cascade (--spec-type): ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, ngram-cache (src/ngram.rs) with --lookup-cache-static / --lookup-cache-dynamic and exact-6-gram priority.
  • RDNA3.5 (gfx1150) / Vulkan optimizations: disable harmful MoE L2 prefetch on gfx11xx/iGPU, honor STRATA_EXPERT_MB with multi-pool fallback, rcp_d wave32 rounding fix (common.glsl), cooperative Q3_K/Q6_K/IQ3_XXS decode shaders, single-submit command-buffer replay, and rel_l2 <= 1e-6 tolerance on sliced attn_prefill QSA mask test for AMD v_wmma 1-ULP rounding.

Verification

  • cargo test --release --lib --bins --tests: 238 passed, 0 failed, 5 ignored on AMD Radeon 890M (RADV STRIX1).

@Vasyl-Trefilov
Vasyl-Trefilov merged commit b99db7d into RustingStudio:main Nov 15, 2026
@Vasyl-Trefilov

Copy link
Copy Markdown
Member

@ybenyedder TYSM for contribute! can you please test it on main to make sure that it's working.

@ybenyedder

Copy link
Copy Markdown
Author

@Vasyl-Trefilov Tested b99db7d on my Radeon 890M (gfx1150). 18/19 test suites pass, only tests/kernels.rs (vulkan_type_matrix and gemv_partial_superblocks_and_q2_0) is still failing because quant_xq runs in wave32 on AMD too, so rcp_d ends up hitting fdiv instead of double division.

I put up a quick PR (#37) to enable double precision rcp_d on AMD wave32 without touching the NVIDIA path. With that, all 19/19 pass on my end.

Vasyl-Trefilov added a commit that referenced this pull request Nov 15, 2026
… drafters, Windows NVRTC lookup

- web UI: terminal-style redesign, message actions, sampler presets, export/import, load options, accent color by model family
- serve: batched decode and batched prefill of queued requests on dense CUDA models; cap of cached CUDA graphs
- serve suggests calibration; no queue limit by default (--max-waiting)
- EAGLE3 speculative decoding, n-gram drafters, --prompt-cache, RDNA3.5 Vulkan work (#36)
- cuda: NVRTC found by our own name list, so Windows finds a CUDA 12.x DLL (#35)
- vulkan: double-precision rcp_d on AMD wave32 devices (#37)
- TQ1_0 and TQ2_0 GGUF types
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants