Skip to content

Reject invalid numeric command line values instead of wrapping or throwing - #1089

Open
sohailshk wants to merge 1 commit into
google:devfrom
sohailshk:fix/args-parse-validation
Open

sohailshk wants to merge 1 commit into
google:devfrom
sohailshk:fix/args-parse-validation

Conversation

@sohailshk

Copy link
Copy Markdown

ArgsBase parsed integer arguments with std::stoi and float arguments with std::stof, and always reported success, so the existing Invalid value for <name> abort was unreachable for numbers.

  • A negative value for a size_t argument wrapped (--top_k=-1, --seq_len=-1).
  • Trailing characters were ignored (--max_generated_tokens=10x parsed as 10).
  • Non-numbers and values outside int range (--seq_len=3000000000) threw an uncaught std::exception and terminated.

This parses with strtoll/strtoull/strtof, requires the whole string to be consumed and the value to fit the target type (negative values for unsigned types are rejected), and returns false so ParseVisitor aborts with Invalid value for <name>. size_t values above INT_MAX are now accepted.

Inputs that were previously accepted by accident now also abort: integer flags with a fraction or exponent (--top_k=40.0 was 40, --seq_len=1e4 was 1), hex (0x10 was 0) and trailing whitespace. No documented usage in the repo uses these forms. (Code reading: --top_k=-1 became SIZE_MAX and only failed after the weights were loaded, at HWY_ASSERT(k <= logits.size()) in TopK.)

Tests: GemmaArgsTest.ParsesValidNumbers and GemmaArgsDeathTest.RejectsInvalidNumbers in gemma/gemma_args_test.cc. On dev (4de902d) the death test fails for every case and the valid-number test aborts on the 3000000000 value; with the change gemma_args_test passes 4/4 (macOS arm64, Apple clang, CMake Release). Not tested: Linux, Windows, Bazel.

…owing

ArgsBase parsed every integer argument with std::stoi and every float with
std::stof, and then always reported success. Consequences:
- A negative value for a size_t argument (--top_k=-1, --seq_len=-1) wrapped
  to a huge value.
- Trailing garbage was ignored (--max_generated_tokens=10x parsed as 10).
- Non-numbers or values outside int range (--seq_len=3000000000 for a size_t)
  threw an uncaught std::exception, so the binary terminated without the
  existing 'Invalid value for' message.

Parse with strtoll/strtoull/strtof, require the whole string to be consumed,
reject out-of-range values for the target type (including negatives for
unsigned types), and return false so ParseVisitor aborts with
'Invalid value for <name>'. Larger size_t values are now accepted.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant