Repository navigation
Conversation
Add twinkle.comm, a standalone (non-default-world) communication group abstraction so checkpoint engines can stop depending on ProcessGroupNCCL, PyHcclCommunicator and ProcessGroupXCCL directly. - base.py: Communicator ABC (create/broadcast/destroy mandatory; all_reduce/barrier/synchronize with defaults; supports() capability query), Rendezvous, ReduceOp, Work - registry.py: lazy backend registry and create_communicator() - backends/gloo.py: ProcessGroupGloo over its own TCPStore (CPU) - Platform.communicator_backend(): nccl (GPU/Metax), hccl (NPU), xccl (XPU), gloo (MPS) - tests/comm: interface/registry contract tests and a multi-process conformance suite parametrized by backend No existing caller is migrated yet; nccl/hccl/xccl backends follow. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Tighten the interface before the nccl/hccl/xccl backends land: - Public methods validate, backends implement hooks (_create, _broadcast, _destroy mandatory; _all_reduce, _barrier optional). - create(): strict option checking against supported_options, explicit timeout, Rendezvous.group_name, device index normalization. - Tensors must be contiguous and on comm.device (no silent copies); zero-element tensors are a no-op on every rank. - destroy(): backends may synchronize inside _destroy; teardown errors are logged, never raised; later operations raise RuntimeError. - barrier(): real readiness point (all_reduce by default). - supports(): explicit op table, unknown names raise. - ReduceOp: AVG, case-insensitive parsing. - Thread-use and current-device rules documented in base.py. - registry: type checks, import errors name the backend, unregister_backend. - gloo: native barrier, host staging for non-CPU devices (MPS), shutdown on destroy. - Platform.communicator_backend(): no base default; GPU declares nccl. - tests: contract tests, accel-marked conformance cases, barrier ordering, threaded double buffering, coexisting groups, rejection cases, join timeout, port-clash retry, import isolation. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
ProcessGroupNCCL over its own TCPStore + PrefixStore(group_name), outside the default world. broadcast/all_reduce map to the raw PG calls; barrier stays on the base all_reduce path because ProcessGroupNCCL.barrier() picks the thread's current device. destroy drains the device, then shuts the PG down. Registered as 'nccl' and added to the conformance matrix (runs on boxes with >= 2 CUDA devices). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
NCCLCheckpointEngine (and DeltaShardedCheckpointEngine through it) now builds its weight-sync group with create_communicator(backend='nccl') instead of a hand-rolled TCPStore + ProcessGroupNCCL: - finalize()/close() destroy the communicator (drain, then PG shutdown) instead of only dropping the reference, so rebuild_group=True no longer leaks a live NCCL communicator per round. - group_name now namespaces the rendezvous store (it was unused). - The init barrier is the communicator's all_reduce barrier on the bound device, not a broadcast on the calling thread's device. - All bucket/flush broadcasts go through one _group_broadcast() seam. XCCLCheckpointEngine overrides it and keeps its own XCCL/relay group; _pg_broadcast moves there, its only remaining user. Tests: a real multi-GPU harness (trainer + 2/3 receivers) for full sync with reused and rebuilt groups, and for delta seed + sparse rounds over NCCL; plus a CPU check of the XCCL seam. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
DeltaShardedCheckpointEngine.prepare() built MasterMetadata without auth_key, while _start_zmq_server() generates one on the master. The subscriber rendezvous probe is HMAC-signed, so receivers (empty key) failed to unpickle it during init_process_group. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Delta receive_weights skips wire probes left queued by the rendezvous (same topic prefix) instead of unpickling them as a manifest. - A rank that fails inside the rendezvous still joins the all_reduce with an error flag, so every rank aborts together rather than blocking until the communicator timeout. - A failed barrier/rendezvous destroys the half-built communicator so the next init_process_group rebuilds it from the metadata it is given. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
NPU.communicator_backend() already returned 'hccl', but the registry only knew gloo/nccl, so create_communicator() raised on Ascend. Add an HCCL backend on torch_npu's native ProcessGroupHCCL over a private TCPStore, with a 'socket_env' option so every rank agrees on the HCCL port layout. The conformance suite gains an hccl/npu case, skipped where the case's own device type is unavailable. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Replace the vLLM StatelessProcessGroup + vllm_ascend PyHcclCommunicator path with the 'hccl' Communicator, so the trainer no longer imports vLLM and full sync gets the shared contract (barrier rendezvous, teardown, option validation). The master publishes its HCCL socket env in MasterMetadata so rollout ranks join with the same port layout; a failed rendezvous barrier destroys the half-built group. Delta HCCL paths are unchanged. Tests cover the group lifecycle with a fake communicator and a multi-process CPU loopback of the real engine over gloo. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
The Ascend full sync has only ever been exercised against a single receiver, which is the narrowest fan-out the broadcast path supports. Add a four-card case where one model rank feeds three sampler ranks over one HCCL group, so the broadcast and the per-receiver finalize are covered at the width real NPU rollouts use. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
A weight sync writes the source tensors only, so every parameter a backend *derives* from them
keeps whatever it computed when the model was built. On Ascend this silently breaks hybrid
attention: vllm-ascend's GDN kernel reads `conv1d.ascend_conv1d_weight`, a kernel-layout copy of
`conv1d.weight` that only `_pack_conv_weights` refreshes ("repack it after every source load" in
`vllm_ascend/ops/gdn.py`), and that repack is wired to `process_weights_after_loading`. A
full-sync therefore left all 18 convolutions convolving with their initial random weights while
every other tensor matched the trainer bit for bit -- transport was verified byte-exact (617/617
tensors), yet the sampler emitted garbage on a Qwen3.5 checkpoint.
Re-run each module's own repack hook once the base weights have landed, and do it for the
speculative drafter as well, whose model is separate. The walk discovers the hook from the live
model, so CUDA (no such hook) and future vllm-ascend builds both work, and a failing hook is
logged without aborting the sync.
Verified on 4x910B2 with vllm-ascend 0.30.0: the Qwen3.5-2B sampler answers correctly after a
sync, "Repacked derived weights for 18 modules" per bucket, and a 3-round train+sync loop now
matches a reference vLLM restarted from each saved checkpoint token for token.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings Ascend NPU up on
twinkle.comm: a newhcclcommunicator backend,HCCLCheckpointEnginemoved off its own process group, and three Ascend-only fixes that a first real 4-card run surfaced.Stacked on #401 (the
twinkle.commcore plus the Gloo/NCCL backends). Until that lands, this branch also carries those six commits; the four HCCL commits are the last ones. Review fromc992b91onwards.hcclbackendHCCLCommunicator(backend namehccl) builds a standaloneProcessGroupHCCLover its own TCPStore, the same shape as thencclbackend:broadcast,all_reduce,barrier,synchronize,destroy, asyncWorkhandles.npudevices, and accepts an optionalsocket_envmap for CANN's socket knobs.ws=2andws=4), which now knowshcclneeds NPUs.Checkpoint engine
HCCLCheckpointEnginenow creates its group throughcreate_communicator(..., backend='hccl', socket_env=...), keeps CANN's rendezvous variables inMasterMetadata.hccl_env, broadcasts on NPU buffers, and destroys a half-built group when the barrier fails. Its delta counterpart is intentionally untouched.Fixes
hcclCommInitRootInfoConfigfailed with error code 1 whenever a second HCCL group came up in the same process, because torch_npu names the communicator afterOptions.group_idand every group defaulted to the same name. The backend now derives one per rendezvous (group_name@host:port). Verified with two coexisting groups, with and without the socket env vars — the earlier "port env var" theory was wrong.conv1d.ascend_conv1d_weight, a kernel-layout copy ofconv1d.weightthat only_pack_conv_weightsrefreshes (its own repack is wired toprocess_weights_after_loading). A full sync therefore left all 18 convolutions convolving with their initial random weights while every other tensor matched the trainer bit for bit — transport was verified byte-exact (617/617 tensors) yet the sampler emitted garbage on a Qwen3.5 checkpoint._load_weightsnow re-runs each module's repack hook once the base weights land, and for the speculative drafter too. The hook is discovered from the live model, so CUDA and future vllm-ascend builds are unaffected, and a hook that raises is logged instead of aborting the sync.Tests
tests/checkpoint_engine/test_hccl_checkpoint_engine.py: a spawn-based harness that runs the real engine on NPU devices and on CPU buffers over gloo —accel(2)loopbacks (fresh group, group rebuild, full sync) plusaccel(4)for a three-receiver fan-out at the width real NPU rollouts use, and unit tests for the metadata and buffer-device paths.tests/comm/:hcclin the conformance and registry suites.tests/sampler/test_vllm_derived_weight_repack.py: the repack helper, its no-op on backends without the hook, its error handling, and its call site (repack strictly after the source tensors land).On 4x Ascend 910B2 (CANN 9.2.1, torch 2.10 / torch_npu 2.10, vLLM 0.30.0 + vllm-ascend 0.30.0, Ray 2.48):
Three environment failures remain, none from this branch: two tests need
caplog/CPU-only torch (the box's twinkle logger writes straight to stderr) and one fails because the image ships a.pthfile that auto-importstorch_npu.E2E (4 cards: FSDP2 trainer on NPU 0-1, vLLM TP=2 rollout on NPU 2-3)
Three rounds of train + full sync, then a reference vLLM restarted from each saved checkpoint.
BLUE-12instead ofBLUE-FALCON-42) because 6 steps x 2 rounds under-train a 1.7B model; the reference loaded from the same checkpoint gives the same answer, so the sync is not implicatedTWINKLE_WEIGHT_SYNC_FINGERPRINT)Sync time on Ascend: 1.9-5.4 s per sync (4.12 GB, HCCL, 2 receivers).