Skip to content

Add tensor filtering before native I/O - #29

Merged
YitaoYuan merged 4 commits into
scitix:mainfrom
Etelis:feat/tensor-filter
Oct 13, 2026
Merged

YitaoYuan merged 4 commits into
scitix:mainfrom
Etelis:feat/tensor-filter

Conversation

@Etelis

@Etelis Etelis commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Add tensor_filter: list[str] to select exact checkpoint names or full-match regexes before native I/O. Keys, buffer sizing, and byte totals reflect the selection. Filtered loads run independently per GPU (process_group=None); page-alignment reads can include neighboring bytes. Filtered copy=False warns about buffer reuse even when tensor bytes fit.

Refs #14. Distributed tensor slicing remains separate.

H100: python -m pytest -q tests/test_layout.py tests/test_io_params.py tests/test_metadata.py tests/test_copy.py — 61 passed, 49 subtests passed, 2 skipped (two GPUs required). Covers exact names with regex characters, regex selections, empty selections, filtered copy warnings, and 1,000 randomized filtered layouts.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
@Etelis
Etelis marked this pull request as ready for review October 12, 2026 06:53
Comment thread instanttensor/_impl.py Resolved
Comment thread instanttensor/_impl.py
keep = tensor_filter is None or tensor_filter(name)
self.tensor_keep.append(keep)
if keep:
self.ordered_tensor_metadatas.append((name, metadata))

@almogtavor almogtavor Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The skip path loader_common.cpp line 382 adds alignment padding for every run of skipped tensors, but the copy=False check at L630 still compares against total_tensor_size: With a filter, each run of kept tensors is read as its own chunk, and each chunk takes extra alignment padding in the GPU buffer (up to about 4 KB). total_tensor_size counts only the tensor bytes, not this padding. So the buffer can be too small to hold all selected tensors at once even when buffer_size >= total_tensor_size. With copy=False the buffer then wraps and overwrites tensors the caller still holds, and this warning doesn’t fire because its check uses total_tensor_size. so add the per-chunk padding to the size used in this check, or always warn when tensor_filter is set with copy=False.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in fa1e850. Filtered copy=False now warns even when buffer_size >= total_tensor_size, covering alignment padding. Added a regression for that case and verified copy=True does not warn. H100 checks: 61 passed, 49 subtests passed, 2 skipped (two GPUs required).

Comment thread csrc/loader_common.cpp
reset_chunk_file(next_file_index, next_tensor_offset);
continue;
}
if (!tensor_keep.empty() && !tensor_keep[source_tensor_id++]) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[perf only--- your output is correct]: each skipped tensor ends the current read, so each run of kept tensors becomes a separate read. If the kept tensors are small and scattered (e.g. only norm weights), that’s many tiny reads instead of a few large ones, and a filtered load can be slower than loading everything. so when the skipped gap is small (e.g. smaller than chunk_size) I think we should read through it instead of starting a new read.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fine for the first version

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping the sparse read boundaries for this first version, as Yitao confirmed above. Small-gap coalescing remains a separate optimization; this PR makes no end-to-end speedup claim.

@robertgshaw2-redhat

Copy link
Copy Markdown

note: this PR is very useful for large scale deployments in vLLM. We have an --enable-ep-weight-filter (used by the safetensors backend). For expert parallel deployments, each rank then only considers the experts mapped to that rank

In large scale deployments (like GLM), EP size can be 32, which allows us to skip loading a significant percentage of weights and dramatically reduce startup time

Co-authored-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com>
@YitaoYuan

Copy link
Copy Markdown
Collaborator

Will merge soon.

@robertgshaw2-redhat

Copy link
Copy Markdown

Will merge soon.

Thank you! Is there a planned next date for pypi release? Would love to pick up this patch in vLLM :)

Thank you for the great work on this project

@YitaoYuan

YitaoYuan commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

I’d suggest making tensor_filter a list[str], where each entry can be an exact tensor name or a regex. This would better align with the planned updates. I can also push an update to your branch, likely later today.

The current changes look correct to me.

I can release a new version immediately after the merge. Looking forward to working closely with the vLLM team.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
@Etelis

Etelis commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Done.

2× H200: python -m pytest -q tests/test_layout.py tests/test_io_params.py tests/test_metadata.py tests/test_copy.py — 63 passed, 53 subtests passed, 0 skipped.

Updated vllm-project/vllm#61167.

Introduce tensor_name_match_mode with exact, glob, and regex support, defaulting to glob. Replace checkpoint-dependent mode inference with a reusable name matcher and preserve None/all and empty-list/none filter semantics.

Use set lookup for literal glob patterns and compile only patterns containing *, ?, or [. Support mixed literal and wildcard lists without changing regex semantics.

Document matching rules and cover literal metacharacters, full-match regexes, invalid inputs, selection metadata, randomized fnmatchcase equivalence, and all modes in GPU layout tests. Update literal-name copy tests to use exact mode.
@YitaoYuan
YitaoYuan merged commit e9b7cba into scitix:main Oct 13, 2026
@YitaoYuan

YitaoYuan commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

@robertgshaw2-redhat @Etelis @almogtavor Merged. Added an optional tensor name matching mode that supports "exact", "glob", and "regex". It defaults to "glob" and automatically falls back to "exact" when safe.

This PR was included in the 0.2.2 release.

@YitaoYuan

Copy link
Copy Markdown
Collaborator

@robertgshaw2-redhat @Etelis @almogtavor I think we can remove the restriction that prevents tensor_filter and multi-GPU loading from being enabled together in the next release. We just need to ensure that all ranks specify the same tensor_filter. If multiple GPUs selectively load the same subset of tensors, using a process_group would likely be faster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants