Repository navigation
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
index out of range error coming from datafusion_row::write::write_field
To Reproduce
It happened when I ran a query against a proprietary data set.
SQL is,
SELECT
tag(column_1),
COUNT(column_1)
FROM
csv
GROUP BY
column_1I haven't been able to reproduce it without that data set.
Expected behavior
It does not panic
Additional context
cargo 1.62.0 (a748cf5a3 2022-06-08)
thread 'tokio-runtime-worker' panicked at 'range end index 153 out of range for slice of length 152', library/core/src/slice/index.rs:73:5
stack backtrace:
0: rust_begin_unwind
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:584:5
1: core::panicking::panic_fmt
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/panicking.rs:142:14
2: core::slice::index::slice_end_index_len_fail_rt
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/slice/index.rs:73:5
3: core::ops::function::FnOnce::call_once
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/ops/function.rs:248:5
4: core::intrinsics::const_eval_select
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/intrinsics.rs:2372:5
5: core::slice::index::slice_end_index_len_fail
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/slice/index.rs:67:9
6: datafusion_row::writer::write_field
7: datafusion_row::writer::write_row
8: <datafusion::physical_plan::aggregates::row_hash::GroupedHashAggregateStreamV2 as futures_core::stream::Stream>::poll_next
9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
10: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
11: tokio::runtime::task::harness::Harness<T,S>::poll
12: std::thread::local::LocalKey<T>::with
13: tokio::runtime::thread_pool::worker::Context::run_task
14: tokio::runtime::thread_pool::worker::Context::run
15: tokio::macros::scoped_tls::ScopedKey<T>::set
16: tokio::runtime::thread_pool::worker::run
17: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
18: tokio::runtime::task::harness::Harness<T,S>::poll
19: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ArrowError(ExternalError(Execution("Join Error: task 15 panicked")))', src/main.rs:47:46
stack backtrace:
0: rust_begin_unwind
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:584:5
1: core::panicking::panic_fmt
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/panicking.rs:142:14
2: core::result::unwrap_failed
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/result.rs:1785:5
3: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
4: std::thread::local::LocalKey<T>::with
5: tokio::park::thread::CachedParkThread::block_on
6: tokio::runtime::Runtime::block_on
7: my_app::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'tokio-runtime-worker' panicked at 'range end index 128 out of range for slice of length 120', library/core/src/slice/index.rs:73:5
stack backtrace:
0: rust_begin_unwind
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:584:5
1: core::panicking::panic_fmt
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/panicking.rs:142:14
2: core::slice::index::slice_end_index_len_fail_rt
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/slice/index.rs:73:5
3: core::ops::function::FnOnce::call_once
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/ops/function.rs:248:5
4: core::intrinsics::const_eval_select
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/intrinsics.rs:2372:5
5: core::slice::index::slice_end_index_len_fail
at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/slice/index.rs:67:9
6: datafusion_row::writer::write_field
7: datafusion_row::writer::write_row
8: <datafusion::physical_plan::aggregates::row_hash::GroupedHashAggregateStreamV2 as futures_core::stream::Stream>::poll_next
9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
10: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
11: tokio::runtime::task::harness::Harness<T,S>::poll
12: std::thread::local::LocalKey<T>::with
13: tokio::runtime::thread_pool::worker::Context::run_task
14: tokio::runtime::thread_pool::worker::Context::run
15: tokio::macros::scoped_tls::ScopedKey<T>::set
16: tokio::runtime::thread_pool::worker::run
17: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
18: tokio::runtime::task::harness::Harness<T,S>::poll
19: tokio::runtime::blocking::pool::Inner::run
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
comphead commentedon Jul 14, 2022
Hi @thomas-k-cameron what is tag function?
thomas-k-cameron commentedon Jul 14, 2022
@comphead
It's my own UDF.
It still generates the same error even without it.
I was able to reproduce the error without the data set I was talking about.
I uploaded the code here.
https://github.com/thomas-k-cameron/df_rs_error_find
comphead commentedon Jul 18, 2022
Thanks for providing data. I was able to reproduce it locally. Looks like #2877 related, but without even having join.
thomas-k-cameron commentedon Jul 18, 2022
Glad to hear that.
Is there anything else that I can do?
comphead commentedon Jul 18, 2022
Nothing more needed for now, I'll try to investigate why its happening. Thanks for reporting such weird thing.
My vision the problem somewhere in hasher, which used both in group by and hash joins.
comphead commentedon Jul 23, 2022
Looks like there are couple of issues in row writer. The interesting thing they occur in very specific scenarios. Still investigating.
thomas-k-cameron commentedon Jul 23, 2022
Lovely. Thanks a lot!
comphead commentedon Jul 26, 2022
Well, maybe it happens before writer, because the same data from CSV and constructed manually works differently. Manual works, CSV failed.