Repository navigation
Conversation
Add twinkle.comm, a standalone (non-default-world) communication group abstraction so checkpoint engines can stop depending on ProcessGroupNCCL, PyHcclCommunicator and ProcessGroupXCCL directly. - base.py: Communicator ABC (create/broadcast/destroy mandatory; all_reduce/barrier/synchronize with defaults; supports() capability query), Rendezvous, ReduceOp, Work - registry.py: lazy backend registry and create_communicator() - backends/gloo.py: ProcessGroupGloo over its own TCPStore (CPU) - Platform.communicator_backend(): nccl (GPU/Metax), hccl (NPU), xccl (XPU), gloo (MPS) - tests/comm: interface/registry contract tests and a multi-process conformance suite parametrized by backend No existing caller is migrated yet; nccl/hccl/xccl backends follow. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Tighten the interface before the nccl/hccl/xccl backends land: - Public methods validate, backends implement hooks (_create, _broadcast, _destroy mandatory; _all_reduce, _barrier optional). - create(): strict option checking against supported_options, explicit timeout, Rendezvous.group_name, device index normalization. - Tensors must be contiguous and on comm.device (no silent copies); zero-element tensors are a no-op on every rank. - destroy(): backends may synchronize inside _destroy; teardown errors are logged, never raised; later operations raise RuntimeError. - barrier(): real readiness point (all_reduce by default). - supports(): explicit op table, unknown names raise. - ReduceOp: AVG, case-insensitive parsing. - Thread-use and current-device rules documented in base.py. - registry: type checks, import errors name the backend, unregister_backend. - gloo: native barrier, host staging for non-CPU devices (MPS), shutdown on destroy. - Platform.communicator_backend(): no base default; GPU declares nccl. - tests: contract tests, accel-marked conformance cases, barrier ordering, threaded double buffering, coexisting groups, rejection cases, join timeout, port-clash retry, import isolation. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
ProcessGroupNCCL over its own TCPStore + PrefixStore(group_name), outside the default world. broadcast/all_reduce map to the raw PG calls; barrier stays on the base all_reduce path because ProcessGroupNCCL.barrier() picks the thread's current device. destroy drains the device, then shuts the PG down. Registered as 'nccl' and added to the conformance matrix (runs on boxes with >= 2 CUDA devices). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
NCCLCheckpointEngine (and DeltaShardedCheckpointEngine through it) now builds its weight-sync group with create_communicator(backend='nccl') instead of a hand-rolled TCPStore + ProcessGroupNCCL: - finalize()/close() destroy the communicator (drain, then PG shutdown) instead of only dropping the reference, so rebuild_group=True no longer leaks a live NCCL communicator per round. - group_name now namespaces the rendezvous store (it was unused). - The init barrier is the communicator's all_reduce barrier on the bound device, not a broadcast on the calling thread's device. - All bucket/flush broadcasts go through one _group_broadcast() seam. XCCLCheckpointEngine overrides it and keeps its own XCCL/relay group; _pg_broadcast moves there, its only remaining user. Tests: a real multi-GPU harness (trainer + 2/3 receivers) for full sync with reused and rebuilt groups, and for delta seed + sparse rounds over NCCL; plus a CPU check of the XCCL seam. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
DeltaShardedCheckpointEngine.prepare() built MasterMetadata without auth_key, while _start_zmq_server() generates one on the master. The subscriber rendezvous probe is HMAC-signed, so receivers (empty key) failed to unpickle it during init_process_group. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Delta receive_weights skips wire probes left queued by the rendezvous (same topic prefix) instead of unpickling them as a manifest. - A rank that fails inside the rendezvous still joins the all_reduce with an error flag, so every rank aborts together rather than blocking until the communicator timeout. - A failed barrier/rendezvous destroys the half-built communicator so the next init_process_group rebuilds it from the metadata it is given. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
kevssim
force-pushed
the
feat/comm-communicator-v5
branch
from
November 19, 2026 10:33
314908e to
da19f06
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
twinkle.comm, a backend-neutral communicator layer for standalone process groups (groups that live outside anytorch.distributeddefault group), and moves the NCCL checkpoint engine onto it.twinkle.commCommunicatorbase class with one contract for every backend:broadcast,all_reduce,barrier,synchronize,destroy, plus asyncWorkhandles. Rank, device and tensor checks live in the base class. Zero-length tensors are a no-op. Calls afterdestroy()are rejected.Rendezvous(host, port, group_name)describes the TCPStore that the group connects through.create_communicator,register_backend,available_backends,default_backend. The default backend comes from the newPlatform.communicator_backend()(GPU/MPS/NPU/XPU each declare theirs; XPU returnsxccl).gloo(CPU) andnccl(a standaloneProcessGroupNCCLbuilt on its own TCPStore).Checkpoint engine
NCCLCheckpointEngine(andDeltaShardedCheckpointEngine, which inherits from it) now hold oneCommunicatorinstead of a raw process group, a TCPStore and_pg_broadcast. Weight broadcasts go through_group_broadcast, whichXCCLCheckpointEngineoverrides with its own group._rendezvous_subscribers(the ZMQ slow-joiner handshake) no longer needs a TCPStore. Each round rank 0 publishes a probe, then all ranksall_reducetwo values: the number of receivers that have consumed a probe and an abort flag. Only rank 0 checks the deadline, so all ranks leave the loop together and no peer is left blocked inside it.dev-swift-v5: a payload-length broadcast that skips the empty final bucket, and the barrier diagnostic log line.Fixes
DeltaShardedCheckpointEngine.prepare()builtMasterMetadatawithoutauth_key, while the master's_start_zmq_server()generates one. The rendezvous probe is HMAC-signed, so delta receivers failed to decode it insideinit_process_group(UnpicklingError). The engine now forwards the key.receive_weightsskips wire probes that the rendezvous leaves queued (same topic prefix) instead of decoding them as a manifest.all_reducewith an error flag, so every rank aborts together instead of blocking until the communicator timeout.init_process_grouprebuilds it from the metadata it is given.Tests
tests/comm/(contract, conformance across backends, registry) andtests/checkpoint_engine/test_nccl_checkpoint_engine.py, a real multi-process NCCL harness covering full sync, delta seed + sparse rounds, group reuse and rebuild, plus unit tests for the rendezvous failure paths and stale-probe skipping.test_nccl_bucket_payload.pynow mocks_group_broadcastinstead of the removed_pg_broadcast.pytest tests/comm tests/checkpoint_engineon 4x RTX 4080 SUPER: 395 passed.E2E (Qwen3.5, vLLM 0.34.0 / torch 2.14 / transformers 5.22.2, Ray, TransformersModel -> NCCL engine -> vLLMSampler)
All rows except the last were run on this branch's head commit.
delta_sharded), Qwen3.5-0.8B, FSDP2Sync time: 1-3 s per sync. The one slow sync (~30 s) comes from the existing accelerate strategy, which keeps the model on CPU until the first forward. It is the same on the base branch and unrelated to this change.