Repository navigation
Add tensor filtering before native I/O - #29
Conversation
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
| keep = tensor_filter is None or tensor_filter(name) | ||
| self.tensor_keep.append(keep) | ||
| if keep: | ||
| self.ordered_tensor_metadatas.append((name, metadata)) |
There was a problem hiding this comment.
The skip path loader_common.cpp line 382 adds alignment padding for every run of skipped tensors, but the copy=False check at L630 still compares against total_tensor_size: With a filter, each run of kept tensors is read as its own chunk, and each chunk takes extra alignment padding in the GPU buffer (up to about 4 KB). total_tensor_size counts only the tensor bytes, not this padding. So the buffer can be too small to hold all selected tensors at once even when buffer_size >= total_tensor_size. With copy=False the buffer then wraps and overwrites tensors the caller still holds, and this warning doesn’t fire because its check uses total_tensor_size. so add the per-chunk padding to the size used in this check, or always warn when tensor_filter is set with copy=False.
There was a problem hiding this comment.
Done in fa1e850. Filtered copy=False now warns even when buffer_size >= total_tensor_size, covering alignment padding. Added a regression for that case and verified copy=True does not warn. H100 checks: 61 passed, 49 subtests passed, 2 skipped (two GPUs required).
| reset_chunk_file(next_file_index, next_tensor_offset); | ||
| continue; | ||
| } | ||
| if (!tensor_keep.empty() && !tensor_keep[source_tensor_id++]) { |
There was a problem hiding this comment.
[perf only--- your output is correct]: each skipped tensor ends the current read, so each run of kept tensors becomes a separate read. If the kept tensors are small and scattered (e.g. only norm weights), that’s many tiny reads instead of a few large ones, and a filtered load can be slower than loading everything. so when the skipped gap is small (e.g. smaller than chunk_size) I think we should read through it instead of starting a new read.
There was a problem hiding this comment.
fine for the first version
There was a problem hiding this comment.
Keeping the sparse read boundaries for this first version, as Yitao confirmed above. Small-gap coalescing remains a separate optimization; this PR makes no end-to-end speedup claim.
|
note: this PR is very useful for large scale deployments in vLLM. We have an --enable-ep-weight-filter (used by the safetensors backend). For expert parallel deployments, each rank then only considers the experts mapped to that rank In large scale deployments (like GLM), EP size can be 32, which allows us to skip loading a significant percentage of weights and dramatically reduce startup time |
Co-authored-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com>
|
Will merge soon. |
Thank you! Is there a planned next date for pypi release? Would love to pick up this patch in vLLM :) Thank you for the great work on this project |
|
I’d suggest making The current changes look correct to me. I can release a new version immediately after the merge. Looking forward to working closely with the vLLM team. |
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
|
Done. 2× H200: Updated vllm-project/vllm#61167. |
Introduce tensor_name_match_mode with exact, glob, and regex support, defaulting to glob. Replace checkpoint-dependent mode inference with a reusable name matcher and preserve None/all and empty-list/none filter semantics. Use set lookup for literal glob patterns and compile only patterns containing *, ?, or [. Support mixed literal and wildcard lists without changing regex semantics. Document matching rules and cover literal metacharacters, full-match regexes, invalid inputs, selection metadata, randomized fnmatchcase equivalence, and all modes in GPU layout tests. Update literal-name copy tests to use exact mode.
|
@robertgshaw2-redhat @Etelis @almogtavor Merged. Added an optional tensor name matching mode that supports This PR was included in the 0.2.2 release. |
|
@robertgshaw2-redhat @Etelis @almogtavor I think we can remove the restriction that prevents |
Add
tensor_filter: list[str]to select exact checkpoint names or full-match regexes before native I/O. Keys, buffer sizing, and byte totals reflect the selection. Filtered loads run independently per GPU (process_group=None); page-alignment reads can include neighboring bytes. Filteredcopy=Falsewarns about buffer reuse even when tensor bytes fit.Refs #14. Distributed tensor slicing remains separate.
H100:
python -m pytest -q tests/test_layout.py tests/test_io_params.py tests/test_metadata.py tests/test_copy.py— 61 passed, 49 subtests passed, 2 skipped (two GPUs required). Covers exact names with regex characters, regex selections, empty selections, filtered copy warnings, and 1,000 randomized filtered layouts.