Skip to content

[C++] Output schema calculated by Substrait consumer for aggregate rel seems incorrect. #34786

Closed

Description

@westonpace

Describe the bug, including details regarding any error messages, version, and platform.

The code calculating the output schema is here:

      FieldVector output_fields;
      output_fields.reserve(key_field_ids.size() + measure_size);
      // extract aggregate fields to output schema
      for (const auto& agg_src_fieldset : agg_src_fieldsets) {
        for (int field : agg_src_fieldset) {
          output_fields.emplace_back(input_schema->field(field));
        }
      }
      // extract key fields to output schema
      for (int key_field_id : key_field_ids) {
        output_fields.emplace_back(input_schema->field(key_field_id));
      }

      std::shared_ptr<Schema> aggregate_schema = schema(std::move(output_fields));

This appears to have two issues:

  • It is inserting the key fields after the measure fields
  • It is inserting measure fields based on the function inputs and not the function outputs

I suspect we are getting away with it in many cases because we are not applying projection / emit after the aggregate. At the very least, we should add some test cases that do this so we can verify the output schema is correct. If there is indeed an issue then we should also fix that.

Component(s)

C++

Activity

  1. changed the title [-][C++] Output schema calculated by Substrait for Aggregate rel seems incorrect.[/-] [+][C++] Output schema calculated by Substrait consumer for aggregate rel seems incorrect.[/+] on Mar 30, 2023
  2. rtpsw commented on Apr 2, 2023

    @rtpsw
    Contributor

    @westonpace, I've looked into this for a while and things are confusing.

    First, I've found that Acero aggregation produces an output schema different than the one you expect here, and in particular one that collects measurements then keys then segment keys. Here are my observations:

    Second, the output-schema-building Arrow Substrait code you noted is trying to collect measurement then keys then segment keys. However, as you noted, it looks like it collects the measurements from the input schema instead of the output schema. I think the code should first construct the output schema (which involves accessing the name field of each Aggregate) and then collect from it.

    Third, the Substrait spec may have a different idea about the expected field order of an aggregation. Assuming so, we should probably fix Acero to use this order, especially since the GroupBy function mentioned in the comment no longer exists.

    Fourth, I suspect that a deeper reason for why we get away with such bugs is that Arrow Substrait generally doesn't care about field names, since Substrait generally doesn't (until the final output), so as long as the field type happens to be right then the test-case passes. Perhaps, besides testing after-projection, we should make the existing test-cases produce different types per field as much as possible.

    cc @icexelloss

  3. icexelloss commented on Apr 2, 2023

    @icexelloss
    Contributor
  4. rtpsw commented on Apr 3, 2023

    @rtpsw
    Contributor

    Can we simply standardize the output column ordering to be segment keys + group by keys + measurements?

    It's doable though not as simple as you might expect - updating the output column ordering would require at least the changing of the many expected outputs explicitly coded in hash_aggregate_test.cc.

  5. rtpsw commented on Apr 3, 2023

    @rtpsw
    Contributor

    Another design issue to notice is that the output schema may (though not usually) differ between planning-time and execution-time. In particular, the output schema depends on the aggregates, which depend on their kernels, which depend on the function registry for function resolution as well as on the exec-context for kernel output-type resolution. Both the function registry and the exec-context, which contains it, may differ (or at least be incompatible for output-schema purposes) between planning-time and execution-time. In particular, at planning-time the current code can only assume a default exec-context. Looks like the best we could do is allow the exec-context to be passed in at planning-time, but this would require changing the signature of a large number of Arrow Substrait functions.

  6. icexelloss commented on Apr 3, 2023

    @icexelloss
    Contributor

    Another design issue to notice is that the output schema may (though not usually) differ between planning-time and execution-time. In particular, the output schema depends on the aggregates, which depend on their kernels, which depend on the function registry for function resolution as well as on the exec-context for kernel output-type resolution. Both the function registry and the exec-context, which contains it, may differ (or at least be incompatible for output-schema purposes) between planning-time and execution-time. In particular, at planning-time the current code can only assume a default exec-context. Looks like the best we could do is allow the exec-context to be passed in at planning-time, but this would require changing the signature of a large number of Arrow Substrait functions.

    Can you give an example of when does the output schema differs in "planning-time" vs "execution'time"?

  7. rtpsw commented on Apr 3, 2023

    @rtpsw
    Contributor

    Can you give an example of when does the output schema differs in "planning-time" vs "execution'time"?

    At execution-time, one can, for example, pass a FunctionRegistry to ExecuteSerializedPlan that is different than the default GetFunctionRegistry() one. The passed registry is allowed to override functions, add kernels, or completely redefine the set of registered functions - making this registry incompatible with the default one used in planning-time for output schema purposes. Granted, this would not be a usual use-case, but it is a supported one.

    My point is that it is a design problem when Acero supports a choice of function registry at execution-time but not at planning-time, because the two choices should be compatible for things to work.

  8. icexelloss commented on Apr 3, 2023

    @icexelloss
    Contributor

    Can you give an example of when does the output schema differs in "planning-time" vs "execution'time"?

    At execution-time, one can, for example, pass a FunctionRegistry to ExecuteSerializedPlan that is different than the default GetFunctionRegistry() one. The passed registry is allowed to override functions, add kernels, or completely redefine the set of registered functions - making this registry incompatible with the default one used in planning-time for output schema purposes. Granted, this would not be a usual use-case, but it is a supported one.

    My point is that it is a design problem when Acero supports a choice of function registry at execution-time but not at planning-time, because the two choices should be compatible for things to work.

    I see - Thanks for the explanation. I am not concerned then. (If someone is passing a Function Registry and changes output schema of existing functions then things certainly will not work, but also there is not really a good reason to do that)

  9. rtpsw commented on Apr 3, 2023

    @rtpsw
    Contributor

    It's doable though not as simple as you might expect - updating the output column ordering would require at least the changing of the many expected outputs explicitly coded in hash_aggregate_test.cc.

    #34551

  10. added a commit that references this issue on Apr 4, 2023
  11. added 3 commits that reference this issue on Apr 5, 2023
  12. added a commit that references this issue on Apr 12, 2023
  13. added this to the 12.0.0 milestone on Apr 12, 2023
  14. added a commit that references this issue on May 11, 2023
  15. added a commit that references this issue on May 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions