Skip to content

Fix TopK returning unaccepted entries when accept_token allows fewer than k - #1088

Open
sohailshk wants to merge 1 commit into
google:devfrom
sohailshk:fix/topk-fewer-accepted-than-k
Open

sohailshk wants to merge 1 commit into
google:devfrom
sohailshk:fix/topk-fewer-accepted-than-k

Conversation

@sohailshk

Copy link
Copy Markdown

TopK collects the tokens accepted by accept_token into a vector and then sorts and unpacks k entries. If fewer than k tokens are accepted (for example RuntimeConfig::top_k = 40 with an accept_token that allows a handful of tokens; reachable through the C++ API, hello_world, run_mmlu and the Python accept= argument), VQSelect/VQSort were called with k larger than the vector and k entries were returned, the surplus being uninitialized memory from the reserved capacity. FusedSoftmaxAndSampleTopK, which only uses token_logits.size() of what it was given, then included those entries in the softmax, so a rejected token could be sampled.

This looks like a regression from #505: before it, the fused sampler used mask = min(k, num_accepted), and #505 kept mask = token_logits.size() while TopK always returns k entries (from reading d854471).

The change clamps to the accepted count, skips VQSelect when every accepted token is kept (Highway's Select requires k < num; it asserts this in debug and sanitizer builds), SampleTopK uses the returned size, and TopK asserts at least one token was accepted.

Test: TestTopKFewerAcceptedThanK in ops/ops_test.cc (k=8, three accepted tokens, negative logits so a zero-valued phantom entry would win). On dev (4de902d) the TopK size check fails on all three targets (8 entries instead of 3) regardless of memory contents; the sampling checks fail only on some iterations, depending on what the unused capacity holds. With the change the full ops_test passes 63/63 (dev has 60; the 3 new ones are the instantiations of this test), also when Highway's sort is built with debug asserts enabled (HWY_IS_DEBUG_BUILD=1; a control with the VQSelect guard removed aborts at vqsort-inl.h:2100, Assert k < num). macOS arm64, Apple clang, CMake Release. Not tested: Linux, Windows, Bazel, debug/sanitizer builds of the whole project.

…than k

TopK sorted and unpacked k entries from a vector that only holds the tokens
accepted by accept_token. When fewer than k tokens were accepted (for example
RuntimeConfig::top_k = 40 with an accept_token that allows a handful of
tokens; reachable through the C++ API, hello_world, run_mmlu or the Python
accept= argument), it called VQSelect/VQSort with k greater than the number of
elements and returned k entries, the surplus being uninitialized memory from
the reserved capacity. FusedSoftmaxAndSampleTopK, which already limits itself
to token_logits.size(), then included those entries in the softmax, so a token
that accept_token had rejected (in practice token 0) could be sampled.

Clamp to the number of accepted tokens and return fewer than k entries in that
case. Skip VQSelect when every accepted token is kept, because Highway's
Select requires k < num. TopK asserts that at least one token was accepted,
and SampleTopK uses the returned size.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant