Skip to content

Hybrid ep paddle - #20

Open
Difers wants to merge 20 commits into
PFCCLab:hybrid-ep-paddlefrom
Difers:hybrid-ep-paddle
Open

Difers wants to merge 20 commits into
PFCCLab:hybrid-ep-paddlefrom
Difers:hybrid-ep-paddle

Conversation

@Difers

@Difers Difers commented Nov 9, 2026

Copy link
Copy Markdown

Update Hybrid EP

Autumn1998 and others added 20 commits February 9, 2026 16:09
* Add hybrid EP token capacity validation

* Pad hybrid EP token capacity to combine chunk
* hybrid_ep: fix scan permute-fusion local-experts sum array sizing

In the permute-fusion path of `scan`, `thread_local_all_block_local_experts_sum`
was declared with `NUM_OF_EXPERTS_PER_RANK - 1` slots, but the load loop
just below writes index `i` for `i in [0, NUM_OF_EXPERTS_PER_RANK)`. When
`i == NUM_OF_EXPERTS_PER_RANK - 1` this overflows the stack array. Bump
the size to `NUM_OF_EXPERTS_PER_RANK` so all written slots are valid.

* nixl: coalesce dispatch prob put with dest-major source layout

The dispatch N2N warp issued one nixlPut per token for the prob slice
because the source layout `[num_tokens][num_of_nodes][prob_per_token]`
interleaved per-destination prob slices on token boundaries, so
per-destination prob bytes weren't contiguous in memory.

Restripe prob into `[num_of_nodes][max_tokens_per_rank][prob_per_token]`
in `dispatch_preprocess` (replacing the existing `cudaMemcpyAsync`), so
the N2N warp issues one coalesced nixlPut per chunk in the dense path
and one per run in the sparse path. The G2S warp's local-node prob
branch is updated to address the dest-major layout (`node_rank *
MAX_TOKENS` base, `prob_per_token` per-token stride). DOCA path is
preserved via `#ifdef USE_NIXL` guards; receive-side layout is
unchanged.

Adds `MAX_NUM_OF_TOKENS_PER_RANK` as a template parameter on the NIXL
N2N warp group.
* fix(build): inherit PyTorch C++ standard for hybrid EP

* test(build): keep standard guard Python 3.8 compatible
…ek-ai#724)

* [HybridEP] Support configurable RoCE settings in DOCA backend

* Fix RoCE configuration edge cases
…pseek-ai#750)

* fix(hybrid_ep): add DMA-BUF registration fallback

* fix(hybrid_ep): harden DMA-BUF registration fallback

* refactor(hybrid_ep): keep GPU registration state in coordinator
Bring the Paddle-compatible HybridEP branch up to upstream 4e890fd
([HybridEP] Fix the fused combine on a non-blocking dispatch, deepseek-ai#766) and
re-apply the PaddlePaddle compat layer on top of the refactored upstream
code.

Notable resolutions:
- Upstream moved the NVL handle exchange into NVLCoordinator; the Paddle
  port (paddle.distributed, `world_size` attribute, explicit host->device
  copy) moved with it, now sized from the tensor instead of a hardcoded
  handle count, which upstream changed from 4/3 to 5/4.
- Kept upstream's dense (int16 topk_idx) routing and its 16B-alignment
  fallback, with the collective switched to paddle.distributed.stream.
- Kept the Paddle JIT file lock and threaded upstream's new
  permute-fusion / token-drop build flags through it.
- indices_to_map keeps upstream's masked scatter_add_, which handles the
  out-of-range topk_idx that ragged dispatch produces; Paddle's compat
  scatter_add_ is put_along_axis_(reduce='add') underneath, so PFCCLab#16's
  intent is preserved and that commit became empty.
- Test suite follows upstream, with the Paddle-only adaptations kept
  (enable_compat, fleet launch, bf16 allclose, uint8 index_select,
  per-node local rank, Kineto skip).

Verified on GB300 (sm_103): deep_ep_cpp and hybrid_ep_cpp both build;
tests/test_hybrid_ep.py passes all 12 correctness checks (BF16 + FP8,
sparse + dense routing, fused + non-fused) on 1 node/4 GPUs, 2 nodes/8
GPUs and 3 nodes/12 GPUs, the latter two over RoCE via the DOCA path.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants