Repository navigation
Allow using 1GB+ data buffers in variable width vectors #721
Description
Describe the enhancement requested
In BaseVariableWidthVector the reallocation logic of the data buffer is to always use the "next power of 2" size instead of the actual desired size (see here). The reallocation fails if the new size exceeds the MAX_BUFFER_SIZE which is Integer.MAX_VALUE by default.
The issue is Integer.MAX_VALUE is just 1 byte less than a power of 2 (231 - 1) which means that exceeding the desired capacity of 1073741824 (1GB), the reallocation would fail. It means, we are unable to get a data buffer larger than 1GB while the default maximum size is almost 2GB.
Activity
lidavidm commentedon Apr 22, 2025
We should clamp the size at 2GB, and we could consider a way to control the preallocation/reallocation behavior (e.g. to start with larger buffers up front)
In general we should be moving towards not having the 2GB limit, but it is baked in for now...
apacheGH-721: Allow using 1GB+ data buffers in variable width vectors
GH-721: Allow using 1GB+ data buffers in variable width vectors (#722)
apacheGH-721: Allow using 1GB+ data buffers in variable width vectors (…