Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
While testing IOX, I hit stack overflowed in build_row_group_predicate with this SQL:
'"SELECT * from restaurant where system > 5.0 and 'tewsbury' != town and system < 7.0 and town = 'reading'"'
Here is the println! of RowGroupPredicateBuilder
Some(
RowGroupPredicateBuilder {
parquet_schema: Schema {
fields: [
Field {
name: "count",
data_type: UInt64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
Field {
name: "system",
data_type: Float64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
Field {
name: "time",
data_type: Timestamp(
Nanosecond,
None,
),
nullable: false,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
Field {
name: "town",
data_type: Dictionary(
Int32,
Utf8,
),
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
],
metadata: {},
},
predicate_expr: BinaryExpr {
left: BinaryExpr {
left: BinaryExpr {
left: BinaryExpr {
left: BinaryExpr {
left: BinaryExpr {
left: BinaryExpr {
left: BinaryExpr {
left: Column {
name: "system_max",
},
op: Gt,
right: Literal {
value: Float64(5),
},
},
op: And,
right: Literal {
value: Boolean(true),
},
},
op: And,
right: BinaryExpr {
left: Column {
name: "system_min",
},
op: Lt,
right: Literal {
value: Float64(7),
},
},
},
op: And,
right: BinaryExpr {
left: BinaryExpr {
left: TryCastExpr {
expr: Column {
name: "town_min",
},
cast_type: Utf8,
},
op: LtEq,
right: Literal {
value: Utf8("reading"),
},
},
op: And,
right: BinaryExpr {
left: Literal {
value: Utf8("reading"),
},
op: LtEq,
right: TryCastExpr {
expr: Column {
name: "town_max",
},
cast_type: Utf8,
},
},
},
},
op: And,
right: BinaryExpr {
left: Column {
name: "system_max",
},
op: Gt,
right: Literal {
value: Float64(5),
},
},
},
op: And,
right: Literal {
value: Boolean(true),
},
},
op: And,
right: BinaryExpr {
left: Column {
name: "system_min",
},
op: Lt,
right: Literal {
value: Float64(7),
},
},
},
op: And,
right: BinaryExpr {
left: BinaryExpr {
left: TryCastExpr {
expr: Column {
name: "town_min",
},
cast_type: Utf8,
},
op: LtEq,
right: Literal {
value: Utf8("reading"),
},
},
op: And,
right: BinaryExpr {
left: Literal {
value: Utf8("reading"),
},
op: LtEq,
right: TryCastExpr {
expr: Column {
name: "town_max",
},
cast_type: Utf8,
},
},
},
},
stat_column_req: [
(
"system",
Max,
Field {
name: "system_max",
data_type: Float64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
),
(
"system",
Min,
Field {
name: "system_min",
data_type: Float64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
),
(
"town",
Min,
Field {
name: "town_min",
data_type: Dictionary(
Int32,
Utf8,
),
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
),
(
"town",
Max,
Field {
name: "town_max",
data_type: Dictionary(
Int32,
Utf8,
),
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
),
],
},
)
I think after this fix #409 merged, the redundant expressions will go away and reduce the size of this predicate but this is an example that we cannot yet handle many expressions in a filter
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
[-]parquet::build_row_grpup_predicate hits stackoverflowed[/-][+]parquet::build_row_gropup_predicate hits stackoverflowed[/+]alamb commentedon May 24, 2021
Some more information:
The test in question is in https://github.com/influxdata/influxdb_iox/pull/1550
When we run it the output is:
When I look in the debugger, here is the stack trace (it isn't obviously some deeply nested problem):
alamb commentedon May 24, 2021
Looks to me like perhaps there is some large intermediate temporary on the stack ?
Igosuki commentedon Sep 12, 2021
Recursion reaches its limits here, and the nesting is only 3 levels deep
alamb commentedon Sep 12, 2021
I wonder if there is some massive structure being placed on the stack somewhere
Dandandan commentedon Sep 12, 2021
Same question here as in #910 Does the error go away when running the tests in release mode?
Igosuki commentedon Sep 15, 2021
@NGA-TRAN see #910 (comment)
NGA-TRAN commentedon Sep 15, 2021
Thanks @Igosuki
alamb commentedon Sep 22, 2021
We are hitting this in IOx now more regularly so I plan to work on a real fix. Assigning to myself
alamb commentedon Sep 23, 2021
I poked a bit at the stack frames in
lldband did some calculations to make sure I was going to optimize in the right place. It does indeed appear to be the case that callingPhysicalExpr::evaluateis consuming ~ 512 kilobytes (yes Kbytes!) per call. No wonder we exhaust the 2M stack size so quickly. Here are the first few frames annotated (frames are SP and then FP (aka the program counter)alamb commentedon Sep 23, 2021
Some observations:
Thus my theory is that because there are so many locals created in
BinaryExpr::evaluate(total, not in use any any one time), the debug build leaves a (unique) space for them all and thus each frame is bloated.I have prototyped fixing the stack error by splitting the
evaluatefunction into several smaller ones (each with fewer variables and thus requiring less stack. I hope to have a PR up shortly.Igosuki commentedon Sep 23, 2021
houqp commentedon Sep 24, 2021
This is nuts, good found @alamb haha
alamb commentedon Sep 24, 2021
Here is a simple reproducer showing the problem:
Dandandan commentedon Sep 24, 2021
Haha - great find @alamb