Skip to content
View AkiYangDev's full-sized avatar

Block or report AkiYangDev

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
AkiYangDev/README.md

Hi, I'm Aki! LLM inference systems, Ascend operators and measured performance.

English · 简体中文

🌟 Merged work   ·   🐋 mini-DeepInfer   ·   📚 Engineering notes

👋 I'm Aki

DeepSeek inference & Ascend NPU engineering

SGLang DeepSeek Ascend 910C Ascend 950

Community fan art of DeepSeek's whale girl

I contribute to SGLang and sgl-kernel-npu, building and debugging DeepSeek inference across Ascend 910C and Ascend 950 platforms.

My upstream work spans MoE / EPLB, prefill/decode state transfer, speculative-decoding statistics, and NPU operator correctness, with reproducible tests and serving measurements.


✅ Merged upstream work

🌟 Physical-storage EPLB on Ascend

SGLang #49701   ✅ Merged · 2026-11-13

Reworked expert migration to move complete physical storage blocks through writable byte views, preserving NZ padding and graph-referenced allocations. Reused TP/HCCL communication and isolated draft warmup/capture from target statistics. Also covered FP4 NZ format 50 on Ascend 950PR.

Earlier merged EPLB work: #47574 added the initial Ascend support (Nov 2); #48751, opened by ch-wan, adopted my core recorder fix (Nov 6). I discovered and reported the bug in issue #48603 and supplied the original fix in #47922.

NZ and packed storage becomes writable expert-byte rows for the migration updater, retaining padding and the original model allocation.

Measured EPLB OFF → ON: output throughput 2,648.89 → 3,627.05 token/s (+36.93%) and mean E2E 50.49 → 36.39 s in the documented DeepSeek-V4-Flash W8A8 PD copying workload. Mean TPOT increased from 8.85 to 10.52 ms/token.

🔬 Experiment setup, acceptance counters & latency

The paired run at 08865d168 used Ascend 910C: 16 physical cards / 32 chips across P+D, TP16/EP16/attention-DP16 per endpoint, DSpark gamma 5 / verify 6, 16,384 input and 1,024 output tokens, concurrency 128, prefix/index caches disabled, and matched actual KV limits.

Throughput counts received generated tokens in the common [60, 360) s window. Latencies use successful requests started inside that window.

Metric EPLB OFF EPLB ON
Output throughput, token/s 2,648.89 3,627.05
Mean TTFT, s 41.43 25.63
Mean TPOT, ms/token 8.85 10.52
Mean E2E, s 50.49 36.39
Acceptance length, tokens/verify 5.988304 5.988304
Measured strict draft acceptance 100% 100%
Correct / proposed draft tokens, latency cohort 658,350 / 658,350 908,010 / 908,010
Full-run successful / failed requests 3,072 / 0 3,072 / 0

All 3,072 paired generated texts match. ON feeds a larger observed decode batch, so the throughput gain comes with a per-request TPOT trade-off. These numbers compare EPLB OFF and ON; the PR separately reports old-ND versus physical-byte migration costs.

More merged work

  • A3 prefill/decode state — SGLang #47934 · Nov 12
    Fixed C4 handoff between different P/D ring sizes with row-sized registration and peer-local addressing.

  • Operator API integration — kernel #1012 · Nov 10
    Added explicit _dsl names across wrappers, registry, tests and bilingual docs while preserving the implementation logic.

  • DeepSeek-V4.1 operators — kernel #964 · Nov 9
    Co-developed seven CANNBot-DSL compute/metadata APIs with ClownBin: dispatch contracts, Torch references, tests and K-tail storage correction.

🔧 Open patches

DeepSeek-V4.1 backend integration on Ascend 950

SGLang #47660 · Open. Adds DeepSeek-V4.1 inference support on Ascend 950: low-ratio KV compression, two-level indexing, mixed-quantization sparse attention, KV/index pools, graph replay, and DSpark integration.

Related integration and upstream alignment: #49005 · Open.

Selected open patches · state transfer, memory, precision & timing (9 PRs)
Area Engineering change PR
PD state Preserve empty components across PP stages; use request-local C4 rows on AMD #47937 / #47938
Gateway lifecycle Cancel pending prefill on early decode failure #49792
Decode memory Bound DeepSeek-V4 SWA storage by request slots #47740
Numerical precision Preserve Softplus accuracy in GDN gating and decode kernel #933
Sparse attention Exclude padded lanes from non-power-of-two TopK reduction kernel #935
Quantization Preserve AWQ fallback dtype; correct GPTQ zero-point offsets #47639 / #47637
Benchmark timing Record non-streaming PD decode handoff time #47772

Open status checked on 2026-11-14. Each link carries the latest upstream state.

📦 More open SGLang patches (12 PRs)
📦 More open NPU operator patches (8 PRs)

🐋 Projects & engineering notes

A runnable teaching inference engine built around DeepInfer's execution design: scheduling, batching, KV-slot reclamation and speculative decoding, from CPU examples to real Qwen3.5 generation on Ascend 910C.

Chinese articles connecting upstream code, reproducible examples and measured behavior. Recent writing:

A cheerful chibi DeepSeek whale girl
Inference, kernels, and a well-reproduced bug. Let's talk 🐋
DeepSeek whale girl · Community fan art

Popular repositories Loading

  1. mini-deepinfer mini-deepinfer Public

    Learn DeepInfer through a small, runnable inference engine. Follow the code on CPU, then run real Qwen3.5 models on Ascend 910C.

    Python 31 1

  2. llm-inference-notes llm-inference-notes Public

    LLM inference engineering: SGLang internals, Ascend deployment, distributed inference, and performance analysis.

    Python 3

  3. AkiYangDev AkiYangDev Public

    Aki Yang | Backend engineering, SGLang, Ascend NPU, and LLM inference.

  4. sglang sglang Public

    Forked from sgl-project/sglang

    SGLang is a high-performance serving framework for large language models and multimodal models.

    Python

  5. sgl-kernel-npu sgl-kernel-npu Public

    Forked from sgl-project/sgl-kernel-npu

    SGLang kernel library for NPU

    C++

  6. ops-transformer ops-transformer Public

    CANN ops-transformer source mirror pinned for CANN 9.2.0 / DSv4.1 kernel builds. Upstream: gitcode.com/cann/ops-transformer

    C++