Skip to content

feat(comm): backend-neutral Communicator (Gloo/NCCL) and NCCL checkpoint engine on it - #401

Open
kevssim wants to merge 6 commits into
modelscope:dev-swift-v5from
kevssim:feat/comm-communicator-v5
Open

kevssim wants to merge 6 commits into
modelscope:dev-swift-v5from
kevssim:feat/comm-communicator-v5

Conversation

@kevssim

@kevssim kevssim commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds twinkle.comm, a backend-neutral communicator layer for standalone process groups (groups that live outside any torch.distributed default group), and moves the NCCL checkpoint engine onto it.

twinkle.comm

  • Communicator base class with one contract for every backend: broadcast, all_reduce, barrier, synchronize, destroy, plus async Work handles. Rank, device and tensor checks live in the base class. Zero-length tensors are a no-op. Calls after destroy() are rejected.
  • Rendezvous(host, port, group_name) describes the TCPStore that the group connects through.
  • Registry: create_communicator, register_backend, available_backends, default_backend. The default backend comes from the new Platform.communicator_backend() (GPU/MPS/NPU/XPU each declare theirs; XPU returns xccl).
  • Backends: gloo (CPU) and nccl (a standalone ProcessGroupNCCL built on its own TCPStore).

Checkpoint engine

  • NCCLCheckpointEngine (and DeltaShardedCheckpointEngine, which inherits from it) now hold one Communicator instead of a raw process group, a TCPStore and _pg_broadcast. Weight broadcasts go through _group_broadcast, which XCCLCheckpointEngine overrides with its own group.
  • _rendezvous_subscribers (the ZMQ slow-joiner handshake) no longer needs a TCPStore. Each round rank 0 publishes a probe, then all ranks all_reduce two values: the number of receivers that have consumed a probe and an abort flag. Only rank 0 checks the deadline, so all ranks leave the loop together and no peer is left blocked inside it.
  • Keeps the behaviour already on dev-swift-v5: a payload-length broadcast that skips the empty final bucket, and the barrier diagnostic log line.

Fixes

  • DeltaShardedCheckpointEngine.prepare() built MasterMetadata without auth_key, while the master's _start_zmq_server() generates one. The rendezvous probe is HMAC-signed, so delta receivers failed to decode it inside init_process_group (UnpicklingError). The engine now forwards the key.
  • The delta receive_weights skips wire probes that the rendezvous leaves queued (same topic prefix) instead of decoding them as a manifest.
  • A rank that fails inside the rendezvous still joins the all_reduce with an error flag, so every rank aborts together instead of blocking until the communicator timeout.
  • A failed barrier or rendezvous destroys the half-built communicator, so the next init_process_group rebuilds it from the metadata it is given.

Tests

  • New: tests/comm/ (contract, conformance across backends, registry) and tests/checkpoint_engine/test_nccl_checkpoint_engine.py, a real multi-process NCCL harness covering full sync, delta seed + sparse rounds, group reuse and rebuild, plus unit tests for the rendezvous failure paths and stale-probe skipping.
  • test_nccl_bucket_payload.py now mocks _group_broadcast instead of the removed _pg_broadcast.
  • pytest tests/comm tests/checkpoint_engine on 4x RTX 4080 SUPER: 395 passed.

E2E (Qwen3.5, vLLM 0.34.0 / torch 2.14 / transformers 5.22.2, Ray, TransformersModel -> NCCL engine -> vLLMSampler)

All rows except the last were run on this branch's head commit.

Scenario Result
LoRA merge, full sync, Qwen3.5-4B 5/5 PASS. The sampler's greedy tokens exactly match a reference vLLM after sync #1 and sync #2, and the trained fact reaches the sampler
Full-parameter FSDP2 full sync, Qwen3.5-4B 4/4 PASS. The sampler exactly matches a reference vLLM loaded from the saved trained checkpoint in rounds 0, 1 and 2, and the trained fact reaches the sampler
Delta sync (delta_sharded), Qwen3.5-0.8B, FSDP2 4/4 PASS. Delta equals full sync in rounds 0, 1 and 2, and the trained fact reaches the sampler
Delta sync with vLLM sleep mode, 0.8B (pre-rebase branch) 4/4 PASS

Sync time: 1-3 s per sync. The one slow sync (~30 s) comes from the existing accelerate strategy, which keeps the model on CPU until the first forward. It is the same on the base branch and unrelated to this change.

kevssim and others added 6 commits November 19, 2026 12:03
Add twinkle.comm, a standalone (non-default-world) communication group
abstraction so checkpoint engines can stop depending on ProcessGroupNCCL,
PyHcclCommunicator and ProcessGroupXCCL directly.

- base.py: Communicator ABC (create/broadcast/destroy mandatory;
  all_reduce/barrier/synchronize with defaults; supports() capability
  query), Rendezvous, ReduceOp, Work
- registry.py: lazy backend registry and create_communicator()
- backends/gloo.py: ProcessGroupGloo over its own TCPStore (CPU)
- Platform.communicator_backend(): nccl (GPU/Metax), hccl (NPU),
  xccl (XPU), gloo (MPS)
- tests/comm: interface/registry contract tests and a multi-process
  conformance suite parametrized by backend

No existing caller is migrated yet; nccl/hccl/xccl backends follow.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Tighten the interface before the nccl/hccl/xccl backends land:

- Public methods validate, backends implement hooks (_create,
  _broadcast, _destroy mandatory; _all_reduce, _barrier optional).
- create(): strict option checking against supported_options, explicit
  timeout, Rendezvous.group_name, device index normalization.
- Tensors must be contiguous and on comm.device (no silent copies);
  zero-element tensors are a no-op on every rank.
- destroy(): backends may synchronize inside _destroy; teardown errors
  are logged, never raised; later operations raise RuntimeError.
- barrier(): real readiness point (all_reduce by default).
- supports(): explicit op table, unknown names raise.
- ReduceOp: AVG, case-insensitive parsing.
- Thread-use and current-device rules documented in base.py.
- registry: type checks, import errors name the backend,
  unregister_backend.
- gloo: native barrier, host staging for non-CPU devices (MPS),
  shutdown on destroy.
- Platform.communicator_backend(): no base default; GPU declares nccl.
- tests: contract tests, accel-marked conformance cases, barrier
  ordering, threaded double buffering, coexisting groups, rejection
  cases, join timeout, port-clash retry, import isolation.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
ProcessGroupNCCL over its own TCPStore + PrefixStore(group_name), outside
the default world. broadcast/all_reduce map to the raw PG calls; barrier
stays on the base all_reduce path because ProcessGroupNCCL.barrier() picks
the thread's current device. destroy drains the device, then shuts the PG
down. Registered as 'nccl' and added to the conformance matrix (runs on
boxes with >= 2 CUDA devices).

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
NCCLCheckpointEngine (and DeltaShardedCheckpointEngine through it) now
builds its weight-sync group with create_communicator(backend='nccl')
instead of a hand-rolled TCPStore + ProcessGroupNCCL:

- finalize()/close() destroy the communicator (drain, then PG shutdown)
  instead of only dropping the reference, so rebuild_group=True no
  longer leaks a live NCCL communicator per round.
- group_name now namespaces the rendezvous store (it was unused).
- The init barrier is the communicator's all_reduce barrier on the
  bound device, not a broadcast on the calling thread's device.
- All bucket/flush broadcasts go through one _group_broadcast() seam.
  XCCLCheckpointEngine overrides it and keeps its own XCCL/relay group;
  _pg_broadcast moves there, its only remaining user.

Tests: a real multi-GPU harness (trainer + 2/3 receivers) for full sync
with reused and rebuilt groups, and for delta seed + sparse rounds over
NCCL; plus a CPU check of the XCCL seam.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
DeltaShardedCheckpointEngine.prepare() built MasterMetadata without
auth_key, while _start_zmq_server() generates one on the master. The
subscriber rendezvous probe is HMAC-signed, so receivers (empty key)
failed to unpickle it during init_process_group.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Delta receive_weights skips wire probes left queued by the rendezvous
  (same topic prefix) instead of unpickling them as a manifest.
- A rank that fails inside the rendezvous still joins the all_reduce with
  an error flag, so every rank aborts together rather than blocking until
  the communicator timeout.
- A failed barrier/rendezvous destroys the half-built communicator so the
  next init_process_group rebuilds it from the metadata it is given.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant