Skip to content

GH-1261: Reject out-of-range dictionary indices in decode - #1262

Open
Arawoof06 wants to merge 1 commit into
apache:mainfrom
Arawoof06:dictionary-index-bounds-check
Open

Arawoof06 wants to merge 1 commit into
apache:mainfrom
Arawoof06:dictionary-index-bounds-check

Conversation

@Arawoof06

Copy link
Copy Markdown
Contributor

DictionaryEncoder.retrieveIndexVector guards each index with indexAsInt > dictionaryCount, but valid indices run 0..dictionaryCount-1, so an index equal to the count reads one slot past the dictionary vector and a negative index from a signed index type is not caught at all; both reach copyValueSafe. The index vector is decoded straight from an IPC payload, so a crafted dictionary-encoded batch reads out of bounds of the dictionary buffers when arrow.enable_unsafe_memory_access is set. Tightening the bound to reject negative indices and indices past the count also covers the list and struct sub-field decoders, which go through the same helper.

Closes #1261.

@jbonofre jbonofre added the bug-fix PRs that fix a big. label Aug 27, 2026
@apache apache deleted a comment from github-actions Bot Aug 27, 2026
@jbonofre jbonofre added this to the 20.0.0 milestone Aug 27, 2026
if (!indices.isNull(i)) {
int indexAsInt = (int) indices.getValueAsLong(i);
if (indexAsInt > dictionaryCount) {
if (indexAsInt < 0 || indexAsInt >= dictionaryCount) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This helper has a caller that doesn't pass a dictionary size. StructSubfieldEncoder.decode calls:

DictionaryEncoder.retrieveIndexVector(indices, transfer, valueCount, 0, valueCount);

where valueCount is the struct's row count. With >= that gives two problems:

  • A 1-row struct whose f0 child holds index 1 into a 2-entry dictionary ["aa", "bb"] decodes on main and now throws Provided dictionary does not contain value for index 1.
  • A 40-row struct with index 39 into a 1-entry dictionary passes 39 >= 40 and reaches copyValueSafe. For a fixed-width dictionary that ends in a raw MemoryUtil.copyMemory in BaseFixedWidthVector.copyFrom.

Could you pass dictionary.getVector().getValueCount() in StructSubfieldEncoder instead? The PR description says the struct decoder is covered by this change, which only holds once that bound is fixed.

Comment on lines 167 to 171
int indexAsInt = (int) indices.getValueAsLong(i);
if (indexAsInt > dictionaryCount) {
if (indexAsInt < 0 || indexAsInt >= dictionaryCount) {
throw new IllegalArgumentException(
"Provided dictionary does not contain value for index " + indexAsInt);
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The narrowing to int happens before the range check, so a 64-bit index can wrap into range. With a BigIntVector index, 4294967296 becomes 0 and -4294967295 becomes 1, and both are accepted. A UInt4 index of 4294967295 is rejected, but the message reports index -1.

Checking the long first fixes both:

Suggested change
int indexAsInt = (int) indices.getValueAsLong(i);
if (indexAsInt > dictionaryCount) {
if (indexAsInt < 0 || indexAsInt >= dictionaryCount) {
throw new IllegalArgumentException(
"Provided dictionary does not contain value for index " + indexAsInt);
}
long index = indices.getValueAsLong(i);
if (index < 0 || index >= dictionaryCount) {
throw new IllegalArgumentException(
"Provided dictionary does not contain value for index " + index);
}
int indexAsInt = (int) index;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix PRs that fix a big.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DictionaryEncoder.decode accepts out-of-range dictionary indices

2 participants