Skip to content

Load files whose largest tensor exceeds the device budget through host memory (opt-in) - #28

Open
meghana-madhyastha wants to merge 2 commits into
scitix:mainfrom
meghana-madhyastha:host-fallback-for-oversized-tensors
Open

meghana-madhyastha wants to merge 2 commits into
scitix:mainfrom
meghana-madhyastha:host-fallback-for-oversized-tensors

Conversation

@meghana-madhyastha

Copy link
Copy Markdown

Problem

The device ring buffer can never be smaller than the largest tensor it carries, so a checkpoint that stores a single tensor larger than the device-memory budget cannot be opened at all. safe_open fails with

RuntimeError: buffer_size (196617056256 B) exceeds device memory budget (54946725888 B)

and the message does not say which tensor is responsible.

A public example is deepseek-ai/DeepSeek-V4.1-Flash: its two engram embedding tables (layers.1.engram.embed.weight, layers.14.engram.embed.weight, F8_E4M3 [384M, 256]) are 91.6 GiB each, each in its own shard (model-00047 / model-00048 of 48). With the default 0.5 budget fraction the recommendation (t[i] + 2*t[i+1]) comes out at 183.1 GiB = the 196617056256 B above, and even the floor (the 91.6 GiB tensor itself) exceeds the budget on every current single GPU. The numbers reproduce offline from the safetensors headers with recommended_buffer_size_for_tensors. No existing knob helps: INSTANTTENSOR_BUFFER_SIZE is enlarged back to the largest tensor and INSTANTTENSOR_MAX_FREE_MEM_USAGE cannot exceed free memory.

Change

  1. Precise error. When a file holds a tensor larger than the budget, safe_open now names the tensor, file and budget, and points at the two ways out.
  2. Opt-in host fallback (host_fallback=True or INSTANTTENSOR_HOST_FALLBACK=1). Files whose largest tensor exceeds the budget are excluded from the native loader and read into host memory with plain buffered reads (64 MiB pieces); their tensors are yielded as CPU tensors that own their memory. Every other file keeps the device path. The file is the unit of exclusion because the native loader requires contiguous byte ranges within a file (compute_layout). Affected files are reported once with a RuntimeWarning. If every file is excluded, the native loader is not opened at all.
  3. Automatic buffer clamp. When the recommendation exceeds the budget but every tensor on the device path still fits, the automatic buffer is shrunk to the budget (with a warning) instead of failing. The recommendation only trades device memory for I/O overlap, so loading still works, it merely overlaps less. An explicit buffer_size above the budget still fails as before.

keys(), offset_keys(), get_tensor_metadata(), the tensors() order and total_tensor_size keep covering every tensor in file order; only ring-buffer sizing (tensor_sizes) is restricted to the device path.

The fallback is opt-in because it changes the placement contract (CPU tensors for the affected files). Consumers whose weight loaders copy into the destination parameter (for example param.copy_(tensor)) work unchanged; the tables in the example above are CPU-resident at runtime anyway, so staging them through the device was never useful for them.

Notes and limitations

  • Fallback reads are plain buffered I/O, so they do not get the native backends' throughput and, in distributed mode, each rank reads the excluded files itself.
  • The host read allocates the full tensor in host memory once (plus one 64 MiB chunk).
  • README and the safe_open docstring document the new argument and environment variable.

Tests

tests/test_host_fallback.py (10 tests) covers the classification, the opt-in and its environment variable, the precise error, the automatic-buffer clamp, the explicit-buffer failure, skipping the native open when all files are host-served, and the host reader (dtype round trip, forced small chunks, truncated files, end-to-end tensors() without a device loader). They run on a CPU-only torch build. tests/test_io_params.py and tests/test_chunk_limit.py have the same outcome before and after on the same box (one test there needs a CUDA build of torch). I could not run the GPU tests here.

🤖 Generated with Claude Code

…t memory (opt-in)

The device ring buffer can never be smaller than the largest tensor it
carries, so a checkpoint that stores one tensor larger than the
device-memory budget cannot be opened at all today: `safe_open` fails with
"buffer_size (...) exceeds device memory budget (...)" and does not say
which tensor is responsible.  DeepSeek-V4.1-Flash is a public example: its
two engram embedding tables (`layers.1.engram.embed.weight` and
`layers.14.engram.embed.weight`, F8_E4M3 [384M, 256]) are 91.6 GiB each and
each sits in its own shard, so the automatic buffer comes out at 183.1 GiB
(196617056256 B) and the model cannot be loaded on any single GPU with the
default 0.5 budget fraction.

This change:

* names the offending tensor, file and budget in the error, and points at
  the two ways out (more device memory, or the new fallback);
* adds `host_fallback` / `INSTANTTENSOR_HOST_FALLBACK=1` (opt-in): files
  whose largest tensor exceeds the budget are excluded from the native
  loader and read into host memory with plain buffered reads, and their
  tensors are yielded as CPU tensors that own their memory.  Every other
  file keeps the device path.  The file is the unit of exclusion because
  the native loader requires contiguous byte ranges within a file.  The
  affected files are reported once with a RuntimeWarning.  If every file is
  excluded the native loader is not opened at all;
* shrinks the automatic buffer to the budget, instead of failing, when the
  recommendation exceeds the budget but every tensor on the device path
  still fits.  The recommendation only trades device memory for I/O
  overlap, so loading still works, it merely overlaps less.  An explicit
  `buffer_size` above the budget still fails as before.

`keys()`, `offset_keys()`, `get_tensor_metadata()`, the `tensors()` order
and `total_tensor_size` keep covering every tensor in file order; only the
ring-buffer sizing (`tensor_sizes`) is restricted to the device path.

Tests: tests/test_host_fallback.py covers the classification, the opt-in
and its environment variable, the precise error, the automatic-buffer
clamp, the explicit-buffer failure, skipping the native open when all
files are host-served, and the host reader (dtype round trip, chunked
reads, truncated files, end-to-end `tensors()` without a device loader).
They run on a CPU-only torch; tests/test_io_params.py and
tests/test_chunk_limit.py are unchanged in outcome on the same box.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@YitaoYuan

YitaoYuan commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Thanks for the contribution — this is a very useful feature. Currently, InstantTensor requires the device buffer to be large enough to hold the largest tensor, which is unacceptable for engram. I think we should add a parameter:

tensor_distribution_config = [
    ("\.engram\.", {"device": "cpu"}),
    ("", {})
]

This is a list of regex patterns that specifies the distribution rule for each tensor. The device field will override the original device. For CPU tensors, we can currently fall back to mmap.

The reason for using this relatively complex approach is that we are implementing some other features that will also use tensor_distribution_config, so implementing it this way is reasonable for future compatibility.

In the future, we may also support sharded loading (not the kind of sharding for TP that other loaders currently support). When enabled by the user, a particularly large tensor can be treated as several sub-tensors, reducing the requirement on device buffer size.

I may continue with further updates based on your branch.

Per review: instead of a file-level host fallback, add an ordered,
regex-keyed placement config so callers choose where each tensor goes,
and so the same structure can carry future per-tensor options.

    tensor_distribution_config = {r"\.engram\.": {"device": "cpu"}}

* Accepted as a dict (insertion order) or a list of (pattern, rule)
  pairs, or as JSON in INSTANTTENSOR_TENSOR_DISTRIBUTION_CONFIG (object
  or list of pairs); an explicit argument wins.  Patterns use re.search
  and the first match applies; unmatched tensors keep the default
  placement.  The only rule key for now is "device": "cpu", or the
  safe_open device itself (a no-op).  Unknown keys, other devices and
  bad patterns are rejected up front.
* CPU tensors are read from a copy-on-write mmap of their file and never
  occupy the ring buffer.  They honour `copy`: owned memory with
  copy=True, zero-copy views of the mapping with copy=False (misaligned
  views are cloned).
* Placement is per tensor, not per file: each maximal run of device
  tensors in a file becomes its own native entry (a file may appear
  more than once), which compute_layout already handles because every
  file-index boundary restarts the chunk at an arbitrary offset.  If no
  tensor is left on the device, the native loader is not opened.
* A device tensor larger than the budget now fails with an error that
  names it and suggests the rule that moves it to the CPU.
* Kept from the previous revision: the automatic buffer is shrunk to
  the budget when every device tensor still fits.

host_fallback / INSTANTTENSOR_HOST_FALLBACK are removed.

Tests: tests/test_tensor_distribution.py (15, CPU-only torch) replaces
tests/test_host_fallback.py and covers config parsing and validation,
first-match semantics, the environment variable, run splitting and the
exact native offsets, the suggested-rule error, the buffer clamp, the
all-CPU case, and CPU reads (dtype round trip, empty tensors, copy vs
zero-copy, the file never being modified, misaligned views).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@meghana-madhyastha

meghana-madhyastha commented Oct 11, 2026 •

Copy link
Copy Markdown
Author

@YitaoYuan , thanks for your review and suggestion. I made some updates based on your suggestion (commit 944b74d). I've reworked the PR around tensor_distribution_config:

  • Ordered (pattern, rule) entries as a dict or a list of pairs, re.search, first match wins, unmatched tensors keep the default placement. Also readable as JSON from INSTANTTENSOR_TENSOR_DISTRIBUTION_CONFIG, since frameworks call safe_open without extra kwargs.
  • {"device": "cpu"} reads the tensor from a copy-on-write mmap of the file; copy=False gives zero-copy views of the mapping.
  • Placement is per tensor: each contiguous run of device tensors in a file becomes its own native entry, so a single .engram. tensor no longer pulls the rest of its shard off the GPU. This relies on compute_layout restarting the chunk at any offset on a file-index boundary; I'd appreciate a check that duplicate filenames in open() are fine on all backends (they get separate fds).
  • Rule keys other than device, and devices other than cpu or the safe_open device, are rejected for now, which leaves room for the extensions you mentioned.
  • host_fallback is gone. I kept the precise error (it now suggests the matching rule) and the automatic-buffer clamp.

I could only run the CPU-side tests; GPU coverage of a mixed file (device runs around a CPU tensor) would be good to have in CI. Happy for you to build on the branch.

@YitaoYuan

Copy link
Copy Markdown
Collaborator

Will merge soon.

@YitaoYuan

Copy link
Copy Markdown
Collaborator

Thanks again for the contribution! We’ve added tensor filtering in #29, which allows tensors to be loaded selectively. I think this is a better fit for your use case: you can exclude tensors intended for the CPU from InstantTensor’s loading path, then load them separately using safetensors or another loader. Another advantage of this approach is that it preserves the existing collective loading semantics.

We also plan to add native support for device="cpu" in InstantTensor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants