Repository navigation
Load files whose largest tensor exceeds the device budget through host memory (opt-in) - #28
Conversation
…t memory (opt-in) The device ring buffer can never be smaller than the largest tensor it carries, so a checkpoint that stores one tensor larger than the device-memory budget cannot be opened at all today: `safe_open` fails with "buffer_size (...) exceeds device memory budget (...)" and does not say which tensor is responsible. DeepSeek-V4.1-Flash is a public example: its two engram embedding tables (`layers.1.engram.embed.weight` and `layers.14.engram.embed.weight`, F8_E4M3 [384M, 256]) are 91.6 GiB each and each sits in its own shard, so the automatic buffer comes out at 183.1 GiB (196617056256 B) and the model cannot be loaded on any single GPU with the default 0.5 budget fraction. This change: * names the offending tensor, file and budget in the error, and points at the two ways out (more device memory, or the new fallback); * adds `host_fallback` / `INSTANTTENSOR_HOST_FALLBACK=1` (opt-in): files whose largest tensor exceeds the budget are excluded from the native loader and read into host memory with plain buffered reads, and their tensors are yielded as CPU tensors that own their memory. Every other file keeps the device path. The file is the unit of exclusion because the native loader requires contiguous byte ranges within a file. The affected files are reported once with a RuntimeWarning. If every file is excluded the native loader is not opened at all; * shrinks the automatic buffer to the budget, instead of failing, when the recommendation exceeds the budget but every tensor on the device path still fits. The recommendation only trades device memory for I/O overlap, so loading still works, it merely overlaps less. An explicit `buffer_size` above the budget still fails as before. `keys()`, `offset_keys()`, `get_tensor_metadata()`, the `tensors()` order and `total_tensor_size` keep covering every tensor in file order; only the ring-buffer sizing (`tensor_sizes`) is restricted to the device path. Tests: tests/test_host_fallback.py covers the classification, the opt-in and its environment variable, the precise error, the automatic-buffer clamp, the explicit-buffer failure, skipping the native open when all files are host-served, and the host reader (dtype round trip, chunked reads, truncated files, end-to-end `tensors()` without a device loader). They run on a CPU-only torch; tests/test_io_params.py and tests/test_chunk_limit.py are unchanged in outcome on the same box. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Thanks for the contribution — this is a very useful feature. Currently, InstantTensor requires the device buffer to be large enough to hold the largest tensor, which is unacceptable for engram. I think we should add a parameter: tensor_distribution_config = [
("\.engram\.", {"device": "cpu"}),
("", {})
]This is a list of regex patterns that specifies the distribution rule for each tensor. The The reason for using this relatively complex approach is that we are implementing some other features that will also use In the future, we may also support sharded loading (not the kind of sharding for TP that other loaders currently support). When enabled by the user, a particularly large tensor can be treated as several sub-tensors, reducing the requirement on device buffer size. I may continue with further updates based on your branch. |
Per review: instead of a file-level host fallback, add an ordered,
regex-keyed placement config so callers choose where each tensor goes,
and so the same structure can carry future per-tensor options.
tensor_distribution_config = {r"\.engram\.": {"device": "cpu"}}
* Accepted as a dict (insertion order) or a list of (pattern, rule)
pairs, or as JSON in INSTANTTENSOR_TENSOR_DISTRIBUTION_CONFIG (object
or list of pairs); an explicit argument wins. Patterns use re.search
and the first match applies; unmatched tensors keep the default
placement. The only rule key for now is "device": "cpu", or the
safe_open device itself (a no-op). Unknown keys, other devices and
bad patterns are rejected up front.
* CPU tensors are read from a copy-on-write mmap of their file and never
occupy the ring buffer. They honour `copy`: owned memory with
copy=True, zero-copy views of the mapping with copy=False (misaligned
views are cloned).
* Placement is per tensor, not per file: each maximal run of device
tensors in a file becomes its own native entry (a file may appear
more than once), which compute_layout already handles because every
file-index boundary restarts the chunk at an arbitrary offset. If no
tensor is left on the device, the native loader is not opened.
* A device tensor larger than the budget now fails with an error that
names it and suggests the rule that moves it to the CPU.
* Kept from the previous revision: the automatic buffer is shrunk to
the budget when every device tensor still fits.
host_fallback / INSTANTTENSOR_HOST_FALLBACK are removed.
Tests: tests/test_tensor_distribution.py (15, CPU-only torch) replaces
tests/test_host_fallback.py and covers config parsing and validation,
first-match semantics, the environment variable, run splitting and the
exact native offsets, the suggested-rule error, the buffer clamp, the
all-CPU case, and CPU reads (dtype round trip, empty tensors, copy vs
zero-copy, the file never being modified, misaligned views).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@YitaoYuan , thanks for your review and suggestion. I made some updates based on your suggestion (commit 944b74d). I've reworked the PR around
I could only run the CPU-side tests; GPU coverage of a mixed file (device runs around a CPU tensor) would be good to have in CI. Happy for you to build on the branch. |
|
Will merge soon. |
|
Thanks again for the contribution! We’ve added tensor filtering in #29, which allows tensors to be loaded selectively. I think this is a better fit for your use case: you can exclude tensors intended for the CPU from InstantTensor’s loading path, then load them separately using We also plan to add native support for |
Problem
The device ring buffer can never be smaller than the largest tensor it carries, so a checkpoint that stores a single tensor larger than the device-memory budget cannot be opened at all.
safe_openfails withand the message does not say which tensor is responsible.
A public example is
deepseek-ai/DeepSeek-V4.1-Flash: its two engram embedding tables (layers.1.engram.embed.weight,layers.14.engram.embed.weight, F8_E4M3[384M, 256]) are 91.6 GiB each, each in its own shard (model-00047/model-00048of 48). With the default 0.5 budget fraction the recommendation (t[i] + 2*t[i+1]) comes out at 183.1 GiB = the 196617056256 B above, and even the floor (the 91.6 GiB tensor itself) exceeds the budget on every current single GPU. The numbers reproduce offline from the safetensors headers withrecommended_buffer_size_for_tensors. No existing knob helps:INSTANTTENSOR_BUFFER_SIZEis enlarged back to the largest tensor andINSTANTTENSOR_MAX_FREE_MEM_USAGEcannot exceed free memory.Change
safe_opennow names the tensor, file and budget, and points at the two ways out.host_fallback=TrueorINSTANTTENSOR_HOST_FALLBACK=1). Files whose largest tensor exceeds the budget are excluded from the native loader and read into host memory with plain buffered reads (64 MiB pieces); their tensors are yielded as CPU tensors that own their memory. Every other file keeps the device path. The file is the unit of exclusion because the native loader requires contiguous byte ranges within a file (compute_layout). Affected files are reported once with aRuntimeWarning. If every file is excluded, the native loader is not opened at all.buffer_sizeabove the budget still fails as before.keys(),offset_keys(),get_tensor_metadata(), thetensors()order andtotal_tensor_sizekeep covering every tensor in file order; only ring-buffer sizing (tensor_sizes) is restricted to the device path.The fallback is opt-in because it changes the placement contract (CPU tensors for the affected files). Consumers whose weight loaders copy into the destination parameter (for example
param.copy_(tensor)) work unchanged; the tables in the example above are CPU-resident at runtime anyway, so staging them through the device was never useful for them.Notes and limitations
safe_opendocstring document the new argument and environment variable.Tests
tests/test_host_fallback.py(10 tests) covers the classification, the opt-in and its environment variable, the precise error, the automatic-buffer clamp, the explicit-buffer failure, skipping the native open when all files are host-served, and the host reader (dtype round trip, forced small chunks, truncated files, end-to-endtensors()without a device loader). They run on a CPU-only torch build.tests/test_io_params.pyandtests/test_chunk_limit.pyhave the same outcome before and after on the same box (one test there needs a CUDA build of torch). I could not run the GPU tests here.🤖 Generated with Claude Code