Skip to content

Load to Intel XPU through a host runtime - #32

Open
kpjeeja wants to merge 2 commits into
scitix:mainfrom
kpjeeja:jeeja/add_support_for_xpu
Open

kpjeeja wants to merge 2 commits into
scitix:mainfrom
kpjeeja:jeeja/add_support_for_xpu

Conversation

@kpjeeja

@kpjeeja kpjeeja commented Nov 19, 2026

Copy link
Copy Markdown

Load to Intel XPU through a host runtime

Motivation

safe_open only accepts CUDA devices today: the native pipeline binds the CUDA or HIP runtime. PyTorch XPU allocates through SYCL's default Level Zero context. Plain Level Zero C code can't get that context's handle without the SYCL C++ API and the oneAPI compiler, so memory from a separate Level Zero binding would be foreign to PyTorch. This PR adds XPU support that keeps PyTorch the only owner of XPU memory and needs no Level Zero headers or oneAPI compiler.

Changes

  • Host runtime (csrc/instant_tensor/dl_binding/cuda_binding.hpp): host_runtime / init_host() fills the CUDA/HIP function table with aligned host allocations, synchronous memcpy, and streams and events that complete immediately. It refuses if CUDA or HIP is already bound in the process.

  • _C.use_host_runtime() (bindings.cpp) selects it before the first open. Under it, DLPack tensors are kDLCPU, device 0 (dlpack_utils.hpp).

  • safe_open(device="xpu") (_internal/loader.py):

    • selects the host runtime;
    • copies each finished tensor to the XPU with .to(device, copy=True), so tensors never alias the ring buffer;
    • synchronizes through the device module;
    • refuses copy=False and process groups with more than one rank for now;
    • skips CUFILE and FGDS.

    CUDA and ROCm paths are unchanged.

  • Memory budget (_internal/resources.py): on XPU, host staging is budgeted as integrated memory over the host's free memory.

  • Capability API (exported from instanttensor/__init__.py), so frameworks can tell whether a build can load to a device:

    instanttensor.supports_device(device_type: str) -> bool         # "xpu" -> True
    instanttensor.supports_process_group(device_type: str) -> bool  # "xpu" -> False (for now)
    The names are a proposal; happy to rename.
    
  • Tests (tests/test_xpu_host_runtime.py):

    • the capability API;
    • bit-exact loads across two files and several dtypes, with ring-buffer reuse;
    • tensors still valid after the context closes;
    • the refusals and the budget.

    The tests run on XPU, or on a host without one with a CPU stand-in for the final copy. They are skipped on CUDA hosts, because the host runtime is process-wide.

  • Docs (separate commit):

    • README.md: Intel XPU added to the requirements, plus an Intel XPU usage section with its limitations and the capability API, and the host-memory budget;
    • docs/source/api.rst: the two capability functions;
    • docs/loader-internals.md: section 10.1 on the host runtime;
    • docs/benchmark.md: the results below.

Overlaps with #19 (CUDA host staging, same function table) and #28 (Python host-memory path); I'll rebase over whichever lands first.

Testing

On 8× Intel Arc Pro B60 (24 GB), Xeon 6980P, torch 2.14.0+xpu:

  • pytest tests/test_xpu_host_runtime.py: 6 passed, with tensors on the XPU.
  • pytest tests: 99 passed, 23 skipped, 2 failed. The failures are test_chunk_size_is_limited_after_page_alignment and test_buffer_environment_is_resolved_once, which already fail on main (07943a6) without this change.
  • End to end in SGLang (--device xpu --load-format instanttensor, Llama-3.1-8B-Instruct, TP1 and TP2): weights are bit-exact against safetensors (/get_weights_by_name, 6 parameters), greedy output is identical, and memory after load is the same.
  • INSTANTTENSOR_BACKEND URING, AIO and BUFFERED all load correctly. CUFILE is refused with CUFILE and FGDS are not available on XPU.

Benchmark

These follow docs/benchmark.md, using tests/test.py adapted for XPU:

  • cuda → xpu and nccl → xccl;
  • copy=True, because copy=False is refused on XPU;
  • no load process group, because supports_process_group("xpu") is False.

Test setting

Versions:

  • safetensors 0.8.0
  • instanttensor: this PR
  • torch 2.14.0+xpu

Hardware and model:

  • Testbed: Intel Xeon 6980P (2 NUMA nodes), 8× Intel Arc Pro B60 24 GB.
  • Model: Qwen3-30B-A3B, first 4 shards (16.0 GB) for 1 GPU and first 8 shards (32.0 GB) for TP=2, as in tests/benchmark_backends.json. A B60 has 24 GB, so the full model can't be placed on one card.
  • fastsafetensors and Run:ai Model Streamer don't support XPU and aren't included.
  • Each number is the median of 3 runs. In every cold-NVMe run, checksums (--checksum) match safetensors; the tmpfs runs were timed without them.

Test on NVMe storage (Crucial P3 4 TB, cold page cache)

The page cache is evicted with posix_fadvise(POSIX_FADV_DONTNEED) before each run.

Model GPU Backend Load Time (s) Throughput (GB/s) Speedup
Qwen3-30B-A3B (4 shards) 1*B60 Safetensors 57.7 0.28 1x
InstantTensor 9.39 1.70 6.1x
InstantTensor (AIO)* 6.68 2.40 8.6x
InstantTensor (URING)* 8.57 1.87 6.7x
InstantTensor (BUFFERED)* 30.3 0.53 1.9x
Qwen3-30B-A3B (8 shards) 2*B60 Safetensors 75.3 0.43 1x
InstantTensor 32.5 0.98 2.3x

*Single run with INSTANTTENSOR_BACKEND set.

This is a consumer QLC drive, and safetensors' cold reads varied a lot (41–134 s for TP=1). InstantTensor's varied much less (9.3–13.5 s).

Test on in-memory file system (tmpfs)

Model GPU Backend Load Time (s) Throughput (GB/s) Speedup
Qwen3-30B-A3B (4 shards) 1*B60 Safetensors 18.7 0.86 1x
InstantTensor 3.35 4.78 5.6x
Qwen3-30B-A3B (8 shards) 2*B60 Safetensors 22.8 1.40 1x
InstantTensor 6.60† 4.85 3.5x

*In multi-GPU loading, the load time is the time until the last (slowest) GPU finishes, and throughput is checkpoint size divided by load time.

†Runs: 15.6, 6.6, 6.4 s (safetensors: 23.1, 22.8, 16.0 s). The host rebooted during the benchmark; the runs at 15.6, 23.1 and 22.8 s came before the reboot, and the rest after it. Both loaders got faster after the reboot. Comparing only runs from the same boot gives 1.5x before the reboot (15.6 s vs 23.1 s) and 2.5x after it (6.4–6.6 s vs 16.0 s).

Why TP=2 gains less

Without a process group, each XPU rank reads the whole checkpoint and copies every full tensor to its device. The planned follow-ups below remove that.

Follow-ups (not in this PR)

  1. Distributed loading over oneCCL (libccl.so via dlopen), with its own communicator; ProcessGroupXCCL has no NCCL communicator to reuse. Then supports_process_group("xpu") becomes True.
  2. Fewer host copies: read directly into the ring buffer, and pin it through PyTorch so device copies can be non_blocking.
  3. Optionally, a small SYCL shim that copies into PyTorch-owned device memory, if profiling shows the host path is the limit.

safe_open accepted only CUDA devices: the native pipeline binds the CUDA
or HIP runtime, and its buffers would be foreign to PyTorch XPU, which
allocates through SYCL's default Level Zero context.

- Add a host runtime to cuda_binding: it fills the same function table
  with aligned host allocations, synchronous memcpy, and streams and
  events that complete immediately. _C.use_host_runtime() selects it
  before the first open; DLPack tensors are then kDLCPU.
- safe_open on xpu selects it and copies each yielded tensor to the
  device with PyTorch, so PyTorch owns all XPU memory. process groups
  and copy=False are refused on XPU for now; CUFILE and FGDS are
  skipped. Once a process has selected the host runtime, CUDA opens
  are refused instead of silently staging in host memory. CUDA and
  ROCm paths are otherwise unchanged.
- Budget XPU staging as integrated memory over the host's free memory.
- Export supports_device() and supports_process_group().
- tests/test_xpu_host_runtime.py: bit-exact loads across files, dtypes
  and ring-buffer reuse, tensor ownership after close, refusals (including
  CUDA after XPU), and the budget; runs on XPU or, without one, with a
  CPU stand-in.
The XPU host runtime shipped without documentation: the README still
listed only CUDA and ROCm, and the capability functions were exported
but missing from the API reference.

- README: list Intel XPU as an experimental platform; note that builds
  need no CUDA/ROCm toolkit, Level Zero headers or oneAPI compiler; add
  an Intel XPU usage section with its limitations (copy=True only, no
  multi-rank process group, no CUFILE/FGDS, process-wide host runtime)
  and the capability API; say NCCL coordination is CUDA/ROCm only; and
  explain the host-memory budget on XPU.
- api.rst: document supports_device and supports_process_group.
- loader-internals.md: add section 10.1 on the host runtime: the
  function table it installs, what stays the same, and the frontend's
  XPU handling.
- benchmark.md: add Intel Arc Pro B60 results for Qwen3-30B-A3B on NVMe
  and tmpfs.
@kpjeeja
kpjeeja force-pushed the jeeja/add_support_for_xpu branch from ca060b3 to 09673f2 Compare November 19, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant