Skip to content

feat(comm): HCCL backend and HCCL checkpoint engine on twinkle.comm - #406

Open
kevssim wants to merge 10 commits into
modelscope:dev-swift-v5from
kevssim:feat/comm-hccl-v5
Open

kevssim wants to merge 10 commits into
modelscope:dev-swift-v5from
kevssim:feat/comm-hccl-v5

Conversation

@kevssim

@kevssim kevssim commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Brings Ascend NPU up on twinkle.comm: a new hccl communicator backend, HCCLCheckpointEngine moved off its own process group, and three Ascend-only fixes that a first real 4-card run surfaced.

Stacked on #401 (the twinkle.comm core plus the Gloo/NCCL backends). Until that lands, this branch also carries those six commits; the four HCCL commits are the last ones. Review from c992b91 onwards.

hccl backend

  • HCCLCommunicator (backend name hccl) builds a standalone ProcessGroupHCCL over its own TCPStore, the same shape as the nccl backend: broadcast, all_reduce, barrier, synchronize, destroy, async Work handles.
  • Rejects non-npu devices, and accepts an optional socket_env map for CANN's socket knobs.
  • Registered in the backend registry and covered by the shared conformance suite (ws=2 and ws=4), which now knows hccl needs NPUs.

Checkpoint engine

  • HCCLCheckpointEngine now creates its group through create_communicator(..., backend='hccl', socket_env=...), keeps CANN's rendezvous variables in MasterMetadata.hccl_env, broadcasts on NPU buffers, and destroys a half-built group when the barrier fails. Its delta counterpart is intentionally untouched.

Fixes

  • Second live group in one process: hcclCommInitRootInfoConfig failed with error code 1 whenever a second HCCL group came up in the same process, because torch_npu names the communicator after Options.group_id and every group defaulted to the same name. The backend now derives one per rendezvous (group_name@host:port). Verified with two coexisting groups, with and without the socket env vars — the earlier "port env var" theory was wrong.
  • Derived weights went stale after a sync: a weight sync writes source tensors only, so parameters a backend derives from them keep their model-build values. vllm-ascend's GDN kernel reads conv1d.ascend_conv1d_weight, a kernel-layout copy of conv1d.weight that only _pack_conv_weights refreshes (its own repack is wired to process_weights_after_loading). A full sync therefore left all 18 convolutions convolving with their initial random weights while every other tensor matched the trainer bit for bit — transport was verified byte-exact (617/617 tensors) yet the sampler emitted garbage on a Qwen3.5 checkpoint. _load_weights now re-runs each module's repack hook once the base weights land, and for the speculative drafter too. The hook is discovered from the live model, so CUDA and future vllm-ascend builds are unaffected, and a hook that raises is logged instead of aborting the sync.

Tests

  • tests/checkpoint_engine/test_hccl_checkpoint_engine.py: a spawn-based harness that runs the real engine on NPU devices and on CPU buffers over gloo — accel(2) loopbacks (fresh group, group rebuild, full sync) plus accel(4) for a three-receiver fan-out at the width real NPU rollouts use, and unit tests for the metadata and buffer-device paths.
  • tests/comm/: hccl in the conformance and registry suites.
  • tests/sampler/test_vllm_derived_weight_repack.py: the repack helper, its no-op on backends without the hook, its error handling, and its call site (repack strictly after the source tensors land).

On 4x Ascend 910B2 (CANN 9.2.1, torch 2.10 / torch_npu 2.10, vLLM 0.30.0 + vllm-ascend 0.30.0, Ray 2.48):

tests/comm + test_hccl_checkpoint_engine.py + test_vllm_derived_weight_repack.py + test_platform_capabilities.py
72 passed, 2 skipped

Three environment failures remain, none from this branch: two tests need caplog/CPU-only torch (the box's twinkle logger writes straight to stderr) and one fails because the image ships a .pth file that auto-imports torch_npu.

E2E (4 cards: FSDP2 trainer on NPU 0-1, vLLM TP=2 rollout on NPU 2-3)

Three rounds of train + full sync, then a reference vLLM restarted from each saved checkpoint.

Scenario Result
Full-parameter FSDP2 full sync, Qwen3.5-2B (hybrid GDN, 617 tensors / 4.12 GB) PASS. Rounds 0, 1 and 2 sampler == reference vLLM token for token, and the trained fact reaches the sampler
Full-parameter FSDP2 full sync, Qwen3-1.7B (text-only closure) Sync verified: rounds 0, 1 and 2 sampler == reference vLLM token for token. The trained-fact check missed (BLUE-12 instead of BLUE-FALCON-42) because 6 steps x 2 rounds under-train a 1.7B model; the reference loaded from the same checkpoint gives the same answer, so the sync is not implicated
Full sync fidelity over HCCL, Qwen3.5-2B 617/617 tensors byte-identical between sender and receiver (TWINKLE_WEIGHT_SYNC_FINGERPRINT)

Sync time on Ascend: 1.9-5.4 s per sync (4.12 GB, HCCL, 2 receivers).

kevssim and others added 10 commits November 19, 2026 12:03
Add twinkle.comm, a standalone (non-default-world) communication group
abstraction so checkpoint engines can stop depending on ProcessGroupNCCL,
PyHcclCommunicator and ProcessGroupXCCL directly.

- base.py: Communicator ABC (create/broadcast/destroy mandatory;
  all_reduce/barrier/synchronize with defaults; supports() capability
  query), Rendezvous, ReduceOp, Work
- registry.py: lazy backend registry and create_communicator()
- backends/gloo.py: ProcessGroupGloo over its own TCPStore (CPU)
- Platform.communicator_backend(): nccl (GPU/Metax), hccl (NPU),
  xccl (XPU), gloo (MPS)
- tests/comm: interface/registry contract tests and a multi-process
  conformance suite parametrized by backend

No existing caller is migrated yet; nccl/hccl/xccl backends follow.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Tighten the interface before the nccl/hccl/xccl backends land:

- Public methods validate, backends implement hooks (_create,
  _broadcast, _destroy mandatory; _all_reduce, _barrier optional).
- create(): strict option checking against supported_options, explicit
  timeout, Rendezvous.group_name, device index normalization.
- Tensors must be contiguous and on comm.device (no silent copies);
  zero-element tensors are a no-op on every rank.
- destroy(): backends may synchronize inside _destroy; teardown errors
  are logged, never raised; later operations raise RuntimeError.
- barrier(): real readiness point (all_reduce by default).
- supports(): explicit op table, unknown names raise.
- ReduceOp: AVG, case-insensitive parsing.
- Thread-use and current-device rules documented in base.py.
- registry: type checks, import errors name the backend,
  unregister_backend.
- gloo: native barrier, host staging for non-CPU devices (MPS),
  shutdown on destroy.
- Platform.communicator_backend(): no base default; GPU declares nccl.
- tests: contract tests, accel-marked conformance cases, barrier
  ordering, threaded double buffering, coexisting groups, rejection
  cases, join timeout, port-clash retry, import isolation.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
ProcessGroupNCCL over its own TCPStore + PrefixStore(group_name), outside
the default world. broadcast/all_reduce map to the raw PG calls; barrier
stays on the base all_reduce path because ProcessGroupNCCL.barrier() picks
the thread's current device. destroy drains the device, then shuts the PG
down. Registered as 'nccl' and added to the conformance matrix (runs on
boxes with >= 2 CUDA devices).

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
NCCLCheckpointEngine (and DeltaShardedCheckpointEngine through it) now
builds its weight-sync group with create_communicator(backend='nccl')
instead of a hand-rolled TCPStore + ProcessGroupNCCL:

- finalize()/close() destroy the communicator (drain, then PG shutdown)
  instead of only dropping the reference, so rebuild_group=True no
  longer leaks a live NCCL communicator per round.
- group_name now namespaces the rendezvous store (it was unused).
- The init barrier is the communicator's all_reduce barrier on the
  bound device, not a broadcast on the calling thread's device.
- All bucket/flush broadcasts go through one _group_broadcast() seam.
  XCCLCheckpointEngine overrides it and keeps its own XCCL/relay group;
  _pg_broadcast moves there, its only remaining user.

Tests: a real multi-GPU harness (trainer + 2/3 receivers) for full sync
with reused and rebuilt groups, and for delta seed + sparse rounds over
NCCL; plus a CPU check of the XCCL seam.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
DeltaShardedCheckpointEngine.prepare() built MasterMetadata without
auth_key, while _start_zmq_server() generates one on the master. The
subscriber rendezvous probe is HMAC-signed, so receivers (empty key)
failed to unpickle it during init_process_group.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Delta receive_weights skips wire probes left queued by the rendezvous
  (same topic prefix) instead of unpickling them as a manifest.
- A rank that fails inside the rendezvous still joins the all_reduce with
  an error flag, so every rank aborts together rather than blocking until
  the communicator timeout.
- A failed barrier/rendezvous destroys the half-built communicator so the
  next init_process_group rebuilds it from the metadata it is given.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
NPU.communicator_backend() already returned 'hccl', but the registry only
knew gloo/nccl, so create_communicator() raised on Ascend. Add an HCCL
backend on torch_npu's native ProcessGroupHCCL over a private TCPStore,
with a 'socket_env' option so every rank agrees on the HCCL port layout.
The conformance suite gains an hccl/npu case, skipped where the case's own
device type is unavailable.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Replace the vLLM StatelessProcessGroup + vllm_ascend PyHcclCommunicator
path with the 'hccl' Communicator, so the trainer no longer imports vLLM
and full sync gets the shared contract (barrier rendezvous, teardown,
option validation). The master publishes its HCCL socket env in
MasterMetadata so rollout ranks join with the same port layout; a failed
rendezvous barrier destroys the half-built group. Delta HCCL paths are
unchanged. Tests cover the group lifecycle with a fake communicator and a
multi-process CPU loopback of the real engine over gloo.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
The Ascend full sync has only ever been exercised against a single receiver, which is the
narrowest fan-out the broadcast path supports. Add a four-card case where one model rank feeds
three sampler ranks over one HCCL group, so the broadcast and the per-receiver finalize are
covered at the width real NPU rollouts use.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
A weight sync writes the source tensors only, so every parameter a backend *derives* from them
keeps whatever it computed when the model was built. On Ascend this silently breaks hybrid
attention: vllm-ascend's GDN kernel reads `conv1d.ascend_conv1d_weight`, a kernel-layout copy of
`conv1d.weight` that only `_pack_conv_weights` refreshes ("repack it after every source load" in
`vllm_ascend/ops/gdn.py`), and that repack is wired to `process_weights_after_loading`. A
full-sync therefore left all 18 convolutions convolving with their initial random weights while
every other tensor matched the trainer bit for bit -- transport was verified byte-exact (617/617
tensors), yet the sampler emitted garbage on a Qwen3.5 checkpoint.

Re-run each module's own repack hook once the base weights have landed, and do it for the
speculative drafter as well, whose model is separate. The walk discovers the hook from the live
model, so CUDA (no such hook) and future vllm-ascend builds both work, and a failing hook is
logged without aborting the sync.

Verified on 4x910B2 with vllm-ascend 0.30.0: the Qwen3.5-2B sampler answers correctly after a
sync, "Repacked derived weights for 18 modules" per bucket, and a 3-round train+sync loop now
matches a reference vLLM restarted from each saved checkpoint token for token.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant