Repository navigation
fix: honor vLLM logits processor transforms in reconstruction - #1853
Conversation
jlarson4
left a comment
There was a problem hiding this comment.
Thanks for the careful fix! Just a few small comments below
| # runs — a per-forward re-fetch clones a d_vocab×d_model tensor to CPU every call. | ||
| self._unembed: tuple[torch.Tensor, Any] | None = None | ||
| self._unembed_probed = False | ||
| self._logits_transform: tuple[float, float | None] | None = None |
There was a problem hiding this comment.
_unembed and _logits_transform now have to be populated together, or _reconstruct_logits trips its assert. That's why the test helpers have to preset _logits_transform. Could the transform live in the _unembed cache so there's a single thing to set and clear?
There was a problem hiding this comment.
Updated in 1623489: _unembed now holds the weight, optional bias, scale, and softcap together. Reconstruction reads that single cache, and teardown clears it in one place; _logits_transform and the separate test-helper initialization are removed. The affected test selection passed locally (165 passed, 18 skipped), along with formatting checks and full-repository mypy; this was not a full-suite run.
| not isinstance(processors[0], Mapping) | ||
| or not {"scale", "soft_cap"} <= processors[0].keys() | ||
| ): | ||
| raise RuntimeError("vLLM logits processor returned malformed metadata.") |
There was a problem hiding this comment.
tl_get_logits_processor already builds this dict and converts both values to float, so the shape check here only guards against the driver's own wire format. Can the value validation sit in the worker?
There was a problem hiding this comment.
Moved the conversion and value validation into tl_get_logits_processor in 1623489, with worker tests for non-finite values, non-positive softcaps, and conversion failures. The driver now trusts the worker wire format and retains the cross-rank agreement check.
| def _reconstruct_logits(self, ln_final: Any) -> torch.Tensor | None: | ||
| """Rebuild real logits from the captured post-weight ln_final: | ||
| ``ln_final @ lm_head.weight.T`` (+ bias, + Gemma-family tanh soft-cap). | ||
| ``ln_final @ lm_head.weight.T`` (+ bias), then the engine's softcap and scale. |
There was a problem hiding this comment.
boot_vllm's docstring in sources/vllm/source.py (line 174) still describes reconstruction as "+ bias, + Gemma soft-cap". Would you be willing to update it to match?
There was a problem hiding this comment.
Updated in 1623489. The boot_vllm docstring now describes unembedding plus bias, followed by the engine softcap and scale, matching the reconstruction path.
|
Looks great @emerardd, thanks for the updates! Approved and merging |
Description
Fix vLLM full-sequence logit reconstruction for architectures whose logits processor applies an output scale. The current driver reconstructs
hidden @ lm_head.weight.T + biasand a config-derived Gemma softcap, but omits the processor's scale. For Cohere/Command-R this dropsconfig.logit_scale; for Granite it drops the effectivelogit_scale / logits_scaling. Probabilities and causal loss are therefore wrong even when a positive scale leaves greedy argmax unchanged.The worker now exposes the actual loaded model's logits processor scale and softcap over a small, plain-data RPC. The reconstruction probe caches those values alongside the unembedding, tolerates absent processors on non-final pipeline stages, verifies agreement across responding ranks, and applies softcap followed by scale after the matmul and bias. This follows the vLLM 0.20.2 processor contract, without architecture-name special cases or assuming local compatibility-fold state applies to remote weights. The architecture setup is visible in Command-R and Granite.
No additional RPC runs during cached reconstruction. When no rank exposes a processor, reconstruction is disabled with an explicit warning instead of guessing an identity transform; the existing last-token fallback cannot claim full-sequence loss support. Unknown/logits-as-input processors and inconsistent or invalid metadata fail loudly. Cache cleanup also releases the transform metadata.
Regression evidence
Offline integration tests construct real tiny HF Cohere and Granite models, capture the real final norm, and exercise actual worker parameter reads plus the actual reconstruction method through a synchronous RPC scaffold. Processor attributes in this scaffold mirror the documented vLLM contract; it is not a live vLLM engine.
On the unchanged base, the non-unit Cohere and Granite scaling cases failed while identity-scale controls passed. The Cohere maximum error was
0.3270536959; the Granite maximum error was0.0767858624. After the fix, reconstructed logits, probabilities, and shifted causal losses match the HF controls. Unit tests additionally cover bias, scale/softcap ordering, cached reads, batched shapes, missing PP stages, replica disagreement, malformed metadata, and scaled logits/loss reachingRemoteBridge.forward(return_type="both")through both single and batched dispatch.Type of change
Validation
test_vllm_logit_scaling.py,test_vllm_driver.py,test_vllm_worker_extension.py,test_vllm_boot.py,test_vllm_plugin.py,test_vllm_internals.py, andtest_inspect_vllm_provider.py.mypy .passed: 387 source files.git diff --checkpassed.The complete test suite was not run locally. No live vLLM/CUDA, compiled graph, tensor-parallel, or pipeline-parallel execution was performed; real engine parity remains a validation boundary for review. The two reported warnings are existing SWIG deprecations. The new missing-processor warning is intentional and tested with
pytest.warns.Checklist
Documentation changes are the reconstruction/probe/RPC docstrings. The warning checkbox is unchecked because the missing-processor diagnostic is intentional. The unit-test checkbox is unchecked because only the affected surface was run, not the complete unit suite.