Repository navigation
Conversation
…owing ArgsBase parsed every integer argument with std::stoi and every float with std::stof, and then always reported success. Consequences: - A negative value for a size_t argument (--top_k=-1, --seq_len=-1) wrapped to a huge value. - Trailing garbage was ignored (--max_generated_tokens=10x parsed as 10). - Non-numbers or values outside int range (--seq_len=3000000000 for a size_t) threw an uncaught std::exception, so the binary terminated without the existing 'Invalid value for' message. Parse with strtoll/strtoull/strtof, require the whole string to be consumed, reject out-of-range values for the target type (including negatives for unsigned types), and return false so ParseVisitor aborts with 'Invalid value for <name>'. Larger size_t values are now accepted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ArgsBaseparsed integer arguments withstd::stoiand float arguments withstd::stof, and always reported success, so the existingInvalid value for <name>abort was unreachable for numbers.size_targument wrapped (--top_k=-1,--seq_len=-1).--max_generated_tokens=10xparsed as 10).intrange (--seq_len=3000000000) threw an uncaughtstd::exceptionand terminated.This parses with
strtoll/strtoull/strtof, requires the whole string to be consumed and the value to fit the target type (negative values for unsigned types are rejected), and returns false soParseVisitoraborts withInvalid value for <name>.size_tvalues aboveINT_MAXare now accepted.Inputs that were previously accepted by accident now also abort: integer flags with a fraction or exponent (
--top_k=40.0was 40,--seq_len=1e4was 1), hex (0x10was 0) and trailing whitespace. No documented usage in the repo uses these forms. (Code reading:--top_k=-1became SIZE_MAX and only failed after the weights were loaded, atHWY_ASSERT(k <= logits.size())inTopK.)Tests:
GemmaArgsTest.ParsesValidNumbersandGemmaArgsDeathTest.RejectsInvalidNumbersingemma/gemma_args_test.cc. Ondev(4de902d) the death test fails for every case and the valid-number test aborts on the 3000000000 value; with the changegemma_args_testpasses 4/4 (macOS arm64, Apple clang, CMake Release). Not tested: Linux, Windows, Bazel.