Conversation
| if (!indices.isNull(i)) { | ||
| int indexAsInt = (int) indices.getValueAsLong(i); | ||
| if (indexAsInt > dictionaryCount) { | ||
| if (indexAsInt < 0 || indexAsInt >= dictionaryCount) { |
There was a problem hiding this comment.
This helper has a caller that doesn't pass a dictionary size. StructSubfieldEncoder.decode calls:
DictionaryEncoder.retrieveIndexVector(indices, transfer, valueCount, 0, valueCount);where valueCount is the struct's row count. With >= that gives two problems:
- A 1-row struct whose
f0child holds index 1 into a 2-entry dictionary["aa", "bb"]decodes on main and now throwsProvided dictionary does not contain value for index 1. - A 40-row struct with index 39 into a 1-entry dictionary passes
39 >= 40and reachescopyValueSafe. For a fixed-width dictionary that ends in a rawMemoryUtil.copyMemoryinBaseFixedWidthVector.copyFrom.
Could you pass dictionary.getVector().getValueCount() in StructSubfieldEncoder instead? The PR description says the struct decoder is covered by this change, which only holds once that bound is fixed.
| int indexAsInt = (int) indices.getValueAsLong(i); | ||
| if (indexAsInt > dictionaryCount) { | ||
| if (indexAsInt < 0 || indexAsInt >= dictionaryCount) { | ||
| throw new IllegalArgumentException( | ||
| "Provided dictionary does not contain value for index " + indexAsInt); | ||
| } |
There was a problem hiding this comment.
The narrowing to int happens before the range check, so a 64-bit index can wrap into range. With a BigIntVector index, 4294967296 becomes 0 and -4294967295 becomes 1, and both are accepted. A UInt4 index of 4294967295 is rejected, but the message reports index -1.
Checking the long first fixes both:
| int indexAsInt = (int) indices.getValueAsLong(i); | |
| if (indexAsInt > dictionaryCount) { | |
| if (indexAsInt < 0 || indexAsInt >= dictionaryCount) { | |
| throw new IllegalArgumentException( | |
| "Provided dictionary does not contain value for index " + indexAsInt); | |
| } | |
| long index = indices.getValueAsLong(i); | |
| if (index < 0 || index >= dictionaryCount) { | |
| throw new IllegalArgumentException( | |
| "Provided dictionary does not contain value for index " + index); | |
| } | |
| int indexAsInt = (int) index; |
DictionaryEncoder.retrieveIndexVector guards each index with
indexAsInt > dictionaryCount, but valid indices run 0..dictionaryCount-1, so an index equal to the count reads one slot past the dictionary vector and a negative index from a signed index type is not caught at all; both reach copyValueSafe. The index vector is decoded straight from an IPC payload, so a crafted dictionary-encoded batch reads out of bounds of the dictionary buffers when arrow.enable_unsafe_memory_access is set. Tightening the bound to reject negative indices and indices past the count also covers the list and struct sub-field decoders, which go through the same helper.Closes #1261.