Repository navigation
Conversation
safe_open accepted only CUDA devices: the native pipeline binds the CUDA or HIP runtime, and its buffers would be foreign to PyTorch XPU, which allocates through SYCL's default Level Zero context. - Add a host runtime to cuda_binding: it fills the same function table with aligned host allocations, synchronous memcpy, and streams and events that complete immediately. _C.use_host_runtime() selects it before the first open; DLPack tensors are then kDLCPU. - safe_open on xpu selects it and copies each yielded tensor to the device with PyTorch, so PyTorch owns all XPU memory. process groups and copy=False are refused on XPU for now; CUFILE and FGDS are skipped. Once a process has selected the host runtime, CUDA opens are refused instead of silently staging in host memory. CUDA and ROCm paths are otherwise unchanged. - Budget XPU staging as integrated memory over the host's free memory. - Export supports_device() and supports_process_group(). - tests/test_xpu_host_runtime.py: bit-exact loads across files, dtypes and ring-buffer reuse, tensor ownership after close, refusals (including CUDA after XPU), and the budget; runs on XPU or, without one, with a CPU stand-in.
The XPU host runtime shipped without documentation: the README still listed only CUDA and ROCm, and the capability functions were exported but missing from the API reference. - README: list Intel XPU as an experimental platform; note that builds need no CUDA/ROCm toolkit, Level Zero headers or oneAPI compiler; add an Intel XPU usage section with its limitations (copy=True only, no multi-rank process group, no CUFILE/FGDS, process-wide host runtime) and the capability API; say NCCL coordination is CUDA/ROCm only; and explain the host-memory budget on XPU. - api.rst: document supports_device and supports_process_group. - loader-internals.md: add section 10.1 on the host runtime: the function table it installs, what stays the same, and the frontend's XPU handling. - benchmark.md: add Intel Arc Pro B60 results for Qwen3-30B-A3B on NVMe and tmpfs.
kpjeeja
force-pushed
the
jeeja/add_support_for_xpu
branch
from
November 19, 2026 18:05
ca060b3 to
09673f2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Load to Intel XPU through a host runtime
Motivation
safe_openonly accepts CUDA devices today: the native pipeline binds the CUDA or HIP runtime. PyTorch XPU allocates through SYCL's default Level Zero context. Plain Level Zero C code can't get that context's handle without the SYCL C++ API and the oneAPI compiler, so memory from a separate Level Zero binding would be foreign to PyTorch. This PR adds XPU support that keeps PyTorch the only owner of XPU memory and needs no Level Zero headers or oneAPI compiler.Changes
Host runtime (
csrc/instant_tensor/dl_binding/cuda_binding.hpp):host_runtime/init_host()fills the CUDA/HIP function table with aligned host allocations, synchronousmemcpy, and streams and events that complete immediately. It refuses if CUDA or HIP is already bound in the process._C.use_host_runtime()(bindings.cpp) selects it before the first open. Under it, DLPack tensors arekDLCPU, device 0 (dlpack_utils.hpp).safe_open(device="xpu")(_internal/loader.py):.to(device, copy=True), so tensors never alias the ring buffer;copy=Falseand process groups with more than one rank for now;CUDA and ROCm paths are unchanged.
Memory budget (
_internal/resources.py): on XPU, host staging is budgeted as integrated memory over the host's free memory.Capability API (exported from
instanttensor/__init__.py), so frameworks can tell whether a build can load to a device:Tests (
tests/test_xpu_host_runtime.py):The tests run on XPU, or on a host without one with a CPU stand-in for the final copy. They are skipped on CUDA hosts, because the host runtime is process-wide.
Docs (separate commit):
README.md: Intel XPU added to the requirements, plus an Intel XPU usage section with its limitations and the capability API, and the host-memory budget;docs/source/api.rst: the two capability functions;docs/loader-internals.md: section 10.1 on the host runtime;docs/benchmark.md: the results below.Overlaps with #19 (CUDA host staging, same function table) and #28 (Python host-memory path); I'll rebase over whichever lands first.
Testing
On 8× Intel Arc Pro B60 (24 GB), Xeon 6980P, torch 2.14.0+xpu:
pytest tests/test_xpu_host_runtime.py: 6 passed, with tensors on the XPU.pytest tests: 99 passed, 23 skipped, 2 failed. The failures aretest_chunk_size_is_limited_after_page_alignmentandtest_buffer_environment_is_resolved_once, which already fail onmain(07943a6) without this change.--device xpu --load-format instanttensor, Llama-3.1-8B-Instruct, TP1 and TP2): weights are bit-exact againstsafetensors(/get_weights_by_name, 6 parameters), greedy output is identical, and memory after load is the same.INSTANTTENSOR_BACKENDURING, AIO and BUFFERED all load correctly. CUFILE is refused withCUFILE and FGDS are not available on XPU.Benchmark
These follow
docs/benchmark.md, usingtests/test.pyadapted for XPU:cuda→xpuandnccl→xccl;copy=True, becausecopy=Falseis refused on XPU;supports_process_group("xpu")is False.Test setting
Versions:
Hardware and model:
tests/benchmark_backends.json. A B60 has 24 GB, so the full model can't be placed on one card.--checksum) match safetensors; the tmpfs runs were timed without them.Test on NVMe storage (Crucial P3 4 TB, cold page cache)
The page cache is evicted with
posix_fadvise(POSIX_FADV_DONTNEED)before each run.*Single run with
INSTANTTENSOR_BACKENDset.This is a consumer QLC drive, and safetensors' cold reads varied a lot (41–134 s for TP=1). InstantTensor's varied much less (9.3–13.5 s).
Test on in-memory file system (tmpfs)
*In multi-GPU loading, the load time is the time until the last (slowest) GPU finishes, and throughput is checkpoint size divided by load time.
†Runs: 15.6, 6.6, 6.4 s (safetensors: 23.1, 22.8, 16.0 s). The host rebooted during the benchmark; the runs at 15.6, 23.1 and 22.8 s came before the reboot, and the rest after it. Both loaders got faster after the reboot. Comparing only runs from the same boot gives 1.5x before the reboot (15.6 s vs 23.1 s) and 2.5x after it (6.4–6.6 s vs 16.0 s).
Why TP=2 gains less
Without a process group, each XPU rank reads the whole checkpoint and copies every full tensor to its device. The planned follow-ups below remove that.
Follow-ups (not in this PR)
libccl.sovia dlopen), with its own communicator;ProcessGroupXCCLhas no NCCL communicator to reuse. Thensupports_process_group("xpu")becomes True.non_blocking.