You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[C++] Crashed at TempStack alloc when use Hashing32::HashBatch independently #40431
Describe the bug, including details regarding any error messages, version, and platform.
The issue is similar to #40007, but they are different.
I want to use the Hashing32::HashBatch api for produce a hash-array for a batch. Although the Hashing32 and Hashing64 are used in join based codes, but they can be used independently.
Like below codes:
auto arr=arrow::ArrayFromJSON(arrow::int32(), "[9,2,6]");
constintbatch_len=arr->length();
arrow::compute::ExecBatchexec_batch({arr}, batch_len);
auto ctx=arrow::compute::default_exec_context();
arrow::util::TempVectorStackstack;
ASSERT_OK(stack.Init(ctx->memory_pool(), batch_len*sizeof(uint32_t))); // I just alloc the stack size as i needed.std::vector<uint32_t>hashes(batch_len);
std::vector<arrow::compute::KeyColumnArray>temp_column_arrays;
ASSERT_OK(arrow::compute::Hashing32::HashBatch(
exec_batch, hashes.data(), temp_column_arrays,
ctx->cpu_info()->hardware_flags(), &stack, 0, batch_len));
auto hash_temp_buf = util::TempVectorHolder<uint32_t>(ctx->stack, max_batch_size);
The holder use the max_batch_size which is 1024 as it's num_elements, it's far more than the temp stack's init buffer_size.
I know that the HashBatch is only used in hash-join or related codes. For join, they have already done line clipping at the upper level, ensuring that each input batch size is less_equal to kMiniBatchLength and the stack size is bigger enough.
But it can be used independently. So maybe we could use the num_rows rather than util::MiniBatch::kMiniBatchLength in HashBatch related apis?
In general, allocating only required size is preferred. So using the num_rows rather than util::MiniBatch::kMiniBatchLength in HashBatch related apis may be better. (Sorry, I can't determine whether the num_rows is the correct size for it right now.)
But it seems that stack.Init(ctx->memory_pool(), batch_len * sizeof(uint32_t)) isn't enough for this case. Because we need at least 3 allocations for HashMultiColumn():
Activity
ZhangHuiGui commentedon Mar 9, 2024
cc @kou PTAL this issue?
kou commentedon Mar 9, 2024
Can you provide a buildable C++ code that reproduces this problem?
ZhangHuiGui commentedon Mar 9, 2024
Of course.
cc @kou, do you think this problem needs to be solved?
kou commentedon Mar 11, 2024
Sorry. I missed this.
Thanks. I could run the code:
In general, allocating only required size is preferred. So using the
num_rowsrather thanutil::MiniBatch::kMiniBatchLengthinHashBatchrelated apis may be better. (Sorry, I can't determine whether thenum_rowsis the correct size for it right now.)But it seems that
stack.Init(ctx->memory_pool(), batch_len * sizeof(uint32_t))isn't enough for this case. Because we need at least 3 allocations forHashMultiColumn():arrow/cpp/src/arrow/compute/key_hash.cc
Lines 387 to 395 in 605f8a7
And we also need 16 bytes metadata:
arrow/cpp/src/arrow/compute/util.cc
Lines 35 to 44 in 605f8a7
Anyway, could you try this?
GH-40431: [C++] Move key_hash/key_map/light_array related files to in…
pitrou commentedon Apr 2, 2024
Issue resolved by pull request 40484
#40484
apacheGH-40431: [C++] Move key_hash/key_map/light_array related files…