Repository navigation
Conversation
…than k TopK sorted and unpacked k entries from a vector that only holds the tokens accepted by accept_token. When fewer than k tokens were accepted (for example RuntimeConfig::top_k = 40 with an accept_token that allows a handful of tokens; reachable through the C++ API, hello_world, run_mmlu or the Python accept= argument), it called VQSelect/VQSort with k greater than the number of elements and returned k entries, the surplus being uninitialized memory from the reserved capacity. FusedSoftmaxAndSampleTopK, which already limits itself to token_logits.size(), then included those entries in the softmax, so a token that accept_token had rejected (in practice token 0) could be sampled. Clamp to the number of accepted tokens and return fewer than k entries in that case. Skip VQSelect when every accepted token is kept, because Highway's Select requires k < num. TopK asserts that at least one token was accepted, and SampleTopK uses the returned size.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TopKcollects the tokens accepted byaccept_tokeninto a vector and then sorts and unpackskentries. If fewer thanktokens are accepted (for exampleRuntimeConfig::top_k = 40with anaccept_tokenthat allows a handful of tokens; reachable through the C++ API, hello_world, run_mmlu and the Pythonaccept=argument),VQSelect/VQSortwere called withklarger than the vector andkentries were returned, the surplus being uninitialized memory from the reserved capacity.FusedSoftmaxAndSampleTopK, which only usestoken_logits.size()of what it was given, then included those entries in the softmax, so a rejected token could be sampled.This looks like a regression from #505: before it, the fused sampler used
mask = min(k, num_accepted), and #505 keptmask = token_logits.size()whileTopKalways returnskentries (from reading d854471).The change clamps to the accepted count, skips
VQSelectwhen every accepted token is kept (Highway'sSelectrequiresk < num; it asserts this in debug and sanitizer builds),SampleTopKuses the returned size, andTopKasserts at least one token was accepted.Test:
TestTopKFewerAcceptedThanKinops/ops_test.cc(k=8, three accepted tokens, negative logits so a zero-valued phantom entry would win). Ondev(4de902d) theTopKsize check fails on all three targets (8 entries instead of 3) regardless of memory contents; the sampling checks fail only on some iterations, depending on what the unused capacity holds. With the change the fullops_testpasses 63/63 (dev has 60; the 3 new ones are the instantiations of this test), also when Highway's sort is built with debug asserts enabled (HWY_IS_DEBUG_BUILD=1; a control with theVQSelectguard removed aborts atvqsort-inl.h:2100,Assert k < num). macOS arm64, Apple clang, CMake Release. Not tested: Linux, Windows, Bazel, debug/sanitizer builds of the whole project.