Repository navigation
Use DataFusionError instead of ArrowError in SendableRecordBatchStream - #5101
Conversation
|
@tustvold please check whenever you have time, the code rotting fast :) |
tustvold
left a comment
There was a problem hiding this comment.
This looks good to me, mostly just some suggestions to use ? instead of map_err(Into::into) in a couple of places.
Given the intrusiveness of this change, I think we should leave this open for a few days in case there are any objections.
| let mut right_indices_builder = UInt32Builder::new(); | ||
| let left_right_indices = match indices_result { | ||
| Err(_) => { | ||
| // TODO why the type of result stream is `Result<T, ArrowError>`, and not the `DataFusionError` |
| .collect::<Result<Vec<_>, ArrowError>>() | ||
| .map_err(Into::<DataFusionError>::into)?; |
There was a problem hiding this comment.
| .collect::<Result<Vec<_>, ArrowError>>() | |
| .map_err(Into::<DataFusionError>::into)?; | |
| .collect::<Result<Vec<_>, ArrowError>>()?; |
| .collect::<Result<Vec<_>, ArrowError>>() | ||
| .map_err(Into::<DataFusionError>::into)? |
There was a problem hiding this comment.
| .collect::<Result<Vec<_>, ArrowError>>() | |
| .map_err(Into::<DataFusionError>::into)? | |
| .collect::<Result<Vec<_>, ArrowError>>()? |
| columns.push(array); | ||
| } | ||
| RecordBatch::try_new(Arc::new(schema.clone()), columns) | ||
| RecordBatch::try_new(Arc::new(schema.clone()), columns).map_err(Into::into) |
There was a problem hiding this comment.
| RecordBatch::try_new(Arc::new(schema.clone()), columns).map_err(Into::into) | |
| Ok(RecordBatch::try_new(Arc::new(schema.clone()), columns)?) |
| .and_then(|b| { | ||
| self.pc_projector | ||
| .project(b, partition_values) | ||
| .map_err(|e| ArrowError::ExternalError(e.into())) |
There was a problem hiding this comment.
Is this needed, project already returns ArrowError AFAICT?
There was a problem hiding this comment.
This is project from DataFusion, it returns DataFusionError
| // apply filter array to record batch | ||
| .and_then(|filter_array| filter_record_batch(batch, filter_array)) | ||
| .and_then(|filter_array| { | ||
| filter_record_batch(batch, filter_array).map_err(Into::into) |
There was a problem hiding this comment.
| filter_record_batch(batch, filter_array).map_err(Into::into) | |
| Ok(filter_record_batch(batch, filter_array)?) |
| ) | ||
| .map_err(Into::<DataFusionError>::into)?; |
There was a problem hiding this comment.
| ) | |
| .map_err(Into::<DataFusionError>::into)?; | |
| )?; |
| let n_generated = columns_to_show[0].len(); | ||
| self.prune_state(n_generated)?; | ||
| RecordBatch::try_new(schema, columns_to_show) | ||
| RecordBatch::try_new(schema, columns_to_show).map_err(Into::into) |
There was a problem hiding this comment.
| RecordBatch::try_new(schema, columns_to_show).map_err(Into::into) | |
| Ok(RecordBatch::try_new(schema, columns_to_show)?) |
| .collect::<Result<Vec<ArrayRef>, ArrowError>>() | ||
| .map_err(Into::<DataFusionError>::into)?; |
There was a problem hiding this comment.
| .collect::<Result<Vec<ArrayRef>, ArrowError>>() | |
| .map_err(Into::<DataFusionError>::into)?; | |
| .collect::<Result<Vec<ArrayRef>, ArrowError>>()?; |
| // calculate window cols | ||
| batch_columns.extend_from_slice(&columns); | ||
| RecordBatch::try_new(self.schema.clone(), batch_columns) | ||
| RecordBatch::try_new(self.schema.clone(), batch_columns).map_err(Into::into) |
There was a problem hiding this comment.
| RecordBatch::try_new(self.schema.clone(), batch_columns).map_err(Into::into) | |
| Ok(RecordBatch::try_new(self.schema.clone(), batch_columns)?) |
|
Thanks @tustvold for your review. Fixed all the comments |
|
cc @liukun4515 , You may also be interested in this. |
|
Also @yahoNanJing and the Ballista team, this may result in chrun |
|
Benchmark runs are scheduled for baseline = 9c8bdfe and contender = 74b05fa. 74b05fa is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Closes #5039.
Rationale for this change
See #5039
What changes are included in this PR?
Replace ArrowError with DataFusionError in DF context
Are these changes tested?
Yes
Are there any user-facing changes?
User will get
DatafusionError::ArrowErrorinsteadarrow::ArrowErrorfor datafusion contexts