Skip to content

Mask ListLayout element reads to the selected lists - #11272

Draft
mhk197 wants to merge 1 commit into
developfrom
mk/list-masked-element-reads
Draft

mhk197 wants to merge 1 commit into
developfrom
mk/list-masked-element-reads

Conversation

@mhk197

@mhk197 mhk197 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Summary

A bounded list read whose selection has holes no longer reads every element between the first and last selected list and filters afterwards. The elements read is masked to the elements the selected lists reference, so the elements reader skips the rest, whole chunks included, and the lists are rebuilt over only what was read.

Changes

  • select_lists and kept_elements (list/reader.rs): the element mask and the kept lists' offsets from the cropped offsets, fetched concurrently with the validity.
  • lists_over: checks the elements read against the last offset before building the list.
  • CountingSegmentSource records the requested segment ids.
microbench (list-layout file) develop this PR MB read
list<u64> (4k × 2,500), raw, half the rows 21.0 ms 2.3 ms 2.5 → 2.5

Testing

  • sparse_mask_skips_element_chunks_between_selected_lists: a mask selecting the first and last list requests two element chunks.
  • random_list_uses_match_memory: random ranges and masks over nullable lists match the in-memory result.

Follow-up

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant