Repository navigation
Conversation
|
Test build #139237 has finished for PR 32753 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
f4ce616 to
64cfc59
Compare
|
Test build #139887 has started for PR 32753 at commit |
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Thank you, @sunchao ! |
|
Test build #139891 has finished for PR 32753 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Thank you for updates, @sunchao ! |
|
Kubernetes integration test starting |
|
Test build #139952 has finished for PR 32753 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Kubernetes integration test status success |
|
Kubernetes integration test status success |
| /** | ||
| * Called at the beginning of reading a new batch. | ||
| * Construct a list of row ranges from the given `rowIndexes`. For example, suppose the | ||
| * `rowIndexes` are `[0, 1, 2, 4, 5, 7, 8, 9]`, it will be converted into 3 row ranges: |
There was a problem hiding this comment.
interesting, does the parquet reader lib give you a big array containing these indexes, or it uses an algorithm to generate the indexes on the fly?
There was a problem hiding this comment.
And how fast/slow the parquet reader lib can generate the indexes?
There was a problem hiding this comment.
It gives you an iterator so yeah generating them on the fly: https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/internal/filter2/columnindex/RowRanges.java#L253. The indexes are generated from Range which is very similar to what we defined here. I'm planning to file a JIRA in parquet-mr to just return the original Ranges so we don't have to do this step in Spark.
There was a problem hiding this comment.
| valuesToReadInBatch -= (newOffset - offset); | ||
| valuesToReadInPage -= (newOffset - offset); | ||
| valuesToReadInPage -= (newRowId - rowId); |
There was a problem hiding this comment.
Should we assert newOffset - offset == newRowId - rowId?
There was a problem hiding this comment.
I think this is not necessarily true: rowId tracks all the values that could be either read or skipped, while offset only tracks value that are read into the result column vector.
| // TIMESTAMP_MILLIS encoded as INT64 can't be lazily decoded as we need to post process | ||
| // the values to add microseconds precision. | ||
| if (column.hasDictionary() || (startOffset == 0 && isLazyDecodingSupported(typeName))) { | ||
| if (column.hasDictionary() || (startRowId == pageFirstRowIndex && |
There was a problem hiding this comment.
Hmm, based on the comment (rowId != 0) below, do we need to update it?
There was a problem hiding this comment.
Oh yeah, I need to update the comment too
| private int readPageV2(DataPageV2 page) throws IOException { | ||
| this.pageFirstRowIndex = page.getFirstRowIndex().orElse(0L); |
There was a problem hiding this comment.
Maybe move to readPage? Looks like readPageV1 and readPageV2 all need it.
There was a problem hiding this comment.
Good point. Let me do that.
| for (int i = 0; i < n; ++i) { | ||
| if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) { | ||
| updater.update(offset + i, values, valueReader); | ||
| int n = Math.min(leftInBatch, Math.min(leftInPage, this.currentCount)); |
There was a problem hiding this comment.
This is different to what I commented (#32753 (comment)) before. This looks more straightforward.
viirya
left a comment
There was a problem hiding this comment.
A few comments. Looks good, otherwise.
|
Kubernetes integration test unable to build dist. exiting with code: 1 |
|
Test build #140382 has finished for PR 32753 at commit
|
| /** Maximum definition level for the Parquet column */ | ||
| final int maxDefinitionLevel; | ||
|
|
||
| /** The current index overall all rows within the column chunk. This is used to check if the |
There was a problem hiding this comment.
Maybe, overall all -> over all?
| ParquetReadState(int maxDefinitionLevel) { | ||
| ParquetReadState(int maxDefinitionLevel, PrimitiveIterator.OfLong rowIndexes) { | ||
| this.maxDefinitionLevel = maxDefinitionLevel; | ||
| this.rowRanges = rowIndexes == null ? null : constructRanges(rowIndexes); |
There was a problem hiding this comment.
If we move this rowIndexes == null ? null ... part into constructRanges method, that will be better because it will protect both this constructor and constructRanges at the same time. For now, constructRanges seems to have the assumption that the argument is not null but there is no assertion for that. WDTY?
There was a problem hiding this comment.
Yes that's a fair point. Will do.
| * @param total total number of values to skip | ||
| * @param valuesReader reader to skip values from | ||
| */ | ||
| void skipValues(int total, VectorizedValuesReader valuesReader); |
There was a problem hiding this comment.
Since this is renamed, please update the following PR description accordingly.
introduced a new API ParquetVectorUpdater.skipBatch which skips a batch of values from a Parquet value reader.
And, maybe, we had better update line 40 in this file.
Skip a batch of ...
There was a problem hiding this comment.
Updated the PR description. Regarding the comment, IMO even though the method name is changed, the comment is still accurate in expressing what the method does.
|
|
||
| // skip the part [rowId, start) | ||
| int toSkip = (int) (start - rowId); | ||
| if (toSkip > 0) { |
There was a problem hiding this comment.
Just a question. We may have two negative value cases.
- start < rowId
- (start - rowId) > Int.MaxValue
Are we considering both? Or, there is no change for case (2)?
There was a problem hiding this comment.
To be safe, I'd like to recommend to move the type casting (int) into inside this if statement. For if (toSkip > 0) { check, we had better use long. If the ranges are protected by line 191 ~ 192, then ignore this comment.
There was a problem hiding this comment.
start must >= rowId because it is defined as long start = Math.max(rangeStart, rowId). Therefore, the case 1 start < rowId will never happen.
The second case, (start - rowId) > Int.MaxValue, can only occur if start is equal to rangeStart. In this case we also know that rangeStart <= rowId + n (from line 183) and n is Math.min(leftInBatch, Math.min(leftInPage, this.currentCount)) which is guaranteed to be within integer range. Therefore, the cast is safe.
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM from my side (with minor questions).
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Merged to master for Apache Spark 3.2.0. Thank you, @sunchao , @viirya , @cloud-fan |
|
Also, cc @gengliangwang since this was the release blocker for Apache Spark 3.2.0. |
|
Test build #140459 has finished for PR 32753 at commit
|
… followed by a dictionary page in a batch ### What changes were proposed in this pull request? This PR fixes two bugs in the vectorized Parquet reader that are hit when, within one batch, a dictionary-encoded data page follows a PLAIN-encoded data page of the same column chunk. 1. **Wrong results from lazy dictionary decoding** (`VectorizedColumnReader.readBatch`). After reading a dictionary-encoded page, the reader attaches the dictionary to the column vector (lazy decoding) only if no non-dictionary values were added to the batch yet. Since SPARK-34859 this was checked with `startRowId == pageFirstRowIndex`, but `ParquetReadState.resetForNewPage` sets `rowId` to `pageFirstRowIndex`, so the check is always true on the first read of a page. The dictionary was then attached over values that a PLAIN page had already written to the batch, and those rows were decoded from dictionary id slots that were never written. This PR restores the check to `startOffset == 0` (no values in the batch yet), so such pages take the existing eager `decodeDictionaryIds` path. This affects flat and nested columns. `startOffset == 0` was the check before SPARK-34859, and it matches that PR's intent: when asked in review [why `readState.rowId == pageFirstRowIndex` is a condition for lazy decoding](#32753 (comment)), the answer was that it is "just to check whether we are at the first iteration of the loop". Rows skipped by column index filtering are not written to the batch, so they don't change `startOffset`. 2. **`ArrayIndexOutOfBoundsException` for repeated columns** (`VectorizedRleValuesReader.readValues`). Values are written at `state.valueOffset`, but `values` / `nulls` were reserved with `reserveAdditional(total)`, i.e. relative to their own `elementsAppended`. For a dictionary-encoded page, `values` is the dictionary ids vector, whose `elementsAppended` does not count the values a PLAIN page already wrote to the batch, so the reservation falls short and `putInts` overflows. This PR reserves `state.valueOffset + total` instead. In all the other cases the two are equal, so the behavior there is unchanged. With off-heap column vectors (`spark.sql.columnVector.offheap.enabled=true`), `OffHeapColumnVector.putInts` does not check bounds, so this overflow writes past the allocated native memory instead of throwing. ### Why are the changes needed? The Parquet format allows a column chunk to mix dictionary-encoded and PLAIN data pages in any order. parquet-mr only falls back from dictionary to PLAIN for good, so files written by Spark do not hit this, but other writers do. We hit this with files written by Snowflake Openflow, where a list element column of a shredded variant has the pages `RLE_DICTIONARY -> PLAIN -> RLE_DICTIONARY`. Reading such files: - silently returns wrong values, e.g. `sum(aggregate(tags, 0L, (a, x) -> a + x))` gives `31228268` instead of `31232468`, or - fails with `java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination index 30504 out of bounds for int[3100]` at `OnHeapColumnVector.putInts` via `VectorizedRleValuesReader.readIntegersRepeated`. Setting `spark.sql.parquet.enableNestedColumnVectorizedReader=false` avoids it only for scans that read nested columns; scans of flat columns stay vectorized and can still return wrong results. The wrong results (1.) exist since Spark 3.2.0 (SPARK-34859). The overflow (2.) can only happen with the nested column vectorized reader, so since Spark 3.3.0. ### Does this PR introduce _any_ user-facing change? Yes, it is a bug fix. Reading Parquet files in which a dictionary-encoded page follows a PLAIN page in the same batch now returns correct results instead of wrong values or an `ArrayIndexOutOfBoundsException`. ### How was this patch tested? Added tests to `ParquetVectorizedSuite` that build column chunks with a given sequence of PLAIN and dictionary-encoded pages and read them with batch sizes that span the pages, with on-heap and off-heap vectors: - primitive type, PLAIN page followed by a dictionary page (wrong results before the fix) - nested type, PLAIN page followed by a dictionary page, one element per row (wrong results before the fix) - nested type, PLAIN page followed by a dictionary page, more values than rows (`ArrayIndexOutOfBoundsException` before the fix) - nested string array with dictionary, PLAIN and dictionary pages, mirroring the files above (wrong results before the fix) - flat and nested columns with column index row ranges that skip the first rows of both pages, so that batches start in the middle of the dictionary page (wrong results before the fix) `checkAnswer` in the suite now also asserts that the reader returns exactly the expected number of rows; before, a reader that stopped early would pass. Also ran `ParquetVectorizedSuite`, `ParquetEncodingSuite`, `ParquetIOSuite`, `ParquetColumnIndexSuite`, `ParquetTypeWideningSuite`, `ParquetV1QuerySuite`, `ParquetV2QuerySuite`, `ParquetFileFormatV1Suite`, `ParquetFileFormatV2Suite`, `ParquetV1SchemaPruningSuite` and `ParquetV2SchemaPruningSuite`. ### Was this patch authored or co-authored using generative AI tooling? Yes, Generated-by: Claude Code (Claude Opus 5.5) Closes #59717 from vinodkc/parquet-plain-then-dict-repro. Authored-by: Vinod KC <vinod.kc.in@gmail.com> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> skip-checks: true
… followed by a dictionary page in a batch ### What changes were proposed in this pull request? This PR fixes two bugs in the vectorized Parquet reader that are hit when, within one batch, a dictionary-encoded data page follows a PLAIN-encoded data page of the same column chunk. 1. **Wrong results from lazy dictionary decoding** (`VectorizedColumnReader.readBatch`). After reading a dictionary-encoded page, the reader attaches the dictionary to the column vector (lazy decoding) only if no non-dictionary values were added to the batch yet. Since SPARK-34859 this was checked with `startRowId == pageFirstRowIndex`, but `ParquetReadState.resetForNewPage` sets `rowId` to `pageFirstRowIndex`, so the check is always true on the first read of a page. The dictionary was then attached over values that a PLAIN page had already written to the batch, and those rows were decoded from dictionary id slots that were never written. This PR restores the check to `startOffset == 0` (no values in the batch yet), so such pages take the existing eager `decodeDictionaryIds` path. This affects flat and nested columns. `startOffset == 0` was the check before SPARK-34859, and it matches that PR's intent: when asked in review [why `readState.rowId == pageFirstRowIndex` is a condition for lazy decoding](#32753 (comment)), the answer was that it is "just to check whether we are at the first iteration of the loop". Rows skipped by column index filtering are not written to the batch, so they don't change `startOffset`. 2. **`ArrayIndexOutOfBoundsException` for repeated columns** (`VectorizedRleValuesReader.readValues`). Values are written at `state.valueOffset`, but `values` / `nulls` were reserved with `reserveAdditional(total)`, i.e. relative to their own `elementsAppended`. For a dictionary-encoded page, `values` is the dictionary ids vector, whose `elementsAppended` does not count the values a PLAIN page already wrote to the batch, so the reservation falls short and `putInts` overflows. This PR reserves `state.valueOffset + total` instead. In all the other cases the two are equal, so the behavior there is unchanged. With off-heap column vectors (`spark.sql.columnVector.offheap.enabled=true`), `OffHeapColumnVector.putInts` does not check bounds, so this overflow writes past the allocated native memory instead of throwing. ### Why are the changes needed? The Parquet format allows a column chunk to mix dictionary-encoded and PLAIN data pages in any order. parquet-mr only falls back from dictionary to PLAIN for good, so files written by Spark do not hit this, but other writers do. We hit this with files written by Snowflake Openflow, where a list element column of a shredded variant has the pages `RLE_DICTIONARY -> PLAIN -> RLE_DICTIONARY`. Reading such files: - silently returns wrong values, e.g. `sum(aggregate(tags, 0L, (a, x) -> a + x))` gives `31228268` instead of `31232468`, or - fails with `java.lang.ArrayIndexOutOfBoundsException: arraycopy: last destination index 30504 out of bounds for int[3100]` at `OnHeapColumnVector.putInts` via `VectorizedRleValuesReader.readIntegersRepeated`. Setting `spark.sql.parquet.enableNestedColumnVectorizedReader=false` avoids it only for scans that read nested columns; scans of flat columns stay vectorized and can still return wrong results. The wrong results (1.) exist since Spark 3.2.0 (SPARK-34859). The overflow (2.) can only happen with the nested column vectorized reader, so since Spark 3.3.0. ### Does this PR introduce _any_ user-facing change? Yes, it is a bug fix. Reading Parquet files in which a dictionary-encoded page follows a PLAIN page in the same batch now returns correct results instead of wrong values or an `ArrayIndexOutOfBoundsException`. ### How was this patch tested? Added tests to `ParquetVectorizedSuite` that build column chunks with a given sequence of PLAIN and dictionary-encoded pages and read them with batch sizes that span the pages, with on-heap and off-heap vectors: - primitive type, PLAIN page followed by a dictionary page (wrong results before the fix) - nested type, PLAIN page followed by a dictionary page, one element per row (wrong results before the fix) - nested type, PLAIN page followed by a dictionary page, more values than rows (`ArrayIndexOutOfBoundsException` before the fix) - nested string array with dictionary, PLAIN and dictionary pages, mirroring the files above (wrong results before the fix) - flat and nested columns with column index row ranges that skip the first rows of both pages, so that batches start in the middle of the dictionary page (wrong results before the fix) `checkAnswer` in the suite now also asserts that the reader returns exactly the expected number of rows; before, a reader that stopped early would pass. Also ran `ParquetVectorizedSuite`, `ParquetEncodingSuite`, `ParquetIOSuite`, `ParquetColumnIndexSuite`, `ParquetTypeWideningSuite`, `ParquetV1QuerySuite`, `ParquetV2QuerySuite`, `ParquetFileFormatV1Suite`, `ParquetFileFormatV2Suite`, `ParquetV1SchemaPruningSuite` and `ParquetV2SchemaPruningSuite`. ### Was this patch authored or co-authored using generative AI tooling? Yes, Generated-by: Claude Code (Claude Opus 5.5) Closes #59717 from vinodkc/parquet-plain-then-dict-repro. Authored-by: Vinod KC <vinod.kc.in@gmail.com> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> (cherry picked from commit 6b576c6) Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> skip-checks: true
What changes were proposed in this pull request?
Make the current vectorized Parquet reader to work with column index introduced in Parquet 1.11. In particular, this PR makes the following changes:
ParquetReadState, track row ranges returned viaPageReadStore.getRowIndexesas well as the first row index for each page viaDataPage.getFirstRowIndex.ParquetVectorUpdater.skipValueswhich skips a batch of values from a Parquet value reader. As part of the process also renamed existingupdateBatchtoreadValues, andupdatetoreadValueto keep the method names consistent.VectorizedValuesReader.skipXXXfor different data types, as well as the implementations. These are useful when the reader knows that the given batch of values can be skipped, for instance, due to the batch is not covered in the row ranges generated by column index filtering.VectorizedRleValuesReaderto handle column index filtering. This is done by comparing the range that is going to be read next within the current RLE/PACKED block (let's call this block range), against the current row range. There are three cases:Why are the changes needed?
Parquet Column Index is a new feature in Parquet 1.11 which allows very efficient filtering on page level (some benchmark numbers can be found here), especially when data is sorted. The feature is largely implemented in parquet-mr (via classes such as
ColumnIndexandColumnIndexFilter). In Spark, the non-vectorized Parquet reader can automatically benefit from the feature after upgrading to Parquet 1.11.x, without any code change. However, the same is not true for vectorized Parquet reader since Spark chose to implement its own logic such as reading Parquet pages, handling definition levels, reading values into columnar batches, etc.Previously, SPARK-26345 / (#31393) updated Spark to only scan pages filtered by column index from parquet-mr side. This is done by calling
ParquetFileReader.readNextFilteredRowGroupandParquetFileReader.getFilteredRecordCountAPI. The implementation, however, only work for a few limited cases: in the scenario where there are multiple columns and their type width are different (e.g.,intandbigint), it could return incorrect result. For this issue, please see SPARK-34859 for a detailed description.In order to fix the above, Spark needs to leverage the API
PageReadStore.getRowIndexesandDataPage.getFirstRowIndex. The former returns the indexes of all rows (note the difference between rows and values: for flat schema there is no difference between the two, but for nested schema they're different) after filtering within a Parquet row group. The latter returns the first row index within a single data page. With the combination of the two, one is able to know which rows/values should be filtered while scanning a Parquet page.Does this PR introduce any user-facing change?
Yes. Now the vectorized Parquet reader should work correctly with column index.
How was this patch tested?
Borrowed tests from #31998 and added a few more tests.