Skip to content

[C++] Potential regression on FieldRef/FieldPath non-flattening Get methods #36892

Closed

Description

@pitrou

Describe the bug, including details regarding any error messages, version, and platform.

I did not pay attention initially, but it seems #35197 introduced a large regression on the wide-dataframe benchmark.

See benchmark results here:
https://conbench.ursa.dev/compare/runs/9cf73ac83f0a44179e6538b2c1c7babd...3d76cb5ffb8849bf8c3ea9b32d08b3b7/

Note the benchmark is creating a dataframe with 10000 columns.

Component(s)

C++

Activity

  1. pitrou commented on Jul 26, 2023

    @pitrou
    MemberAuthor

    @benibus Would you have time to take a look at this?
    It would be nice to add benchmarks for the non-flattening methods. They will also help measuring the slowdown before/after that PR.

  2. raulcd commented on Jul 28, 2023

    @raulcd
    Member

    @benibus @westonpace I was planning on waiting a couple days to create the next RC (Tuesday?) to see if this was able to make it to 13.0.0. If this sounds totally unreasonable let me know and I can create the new RC sooner as discussed on Zulip we might want to ship 13.0.0 with this known issue and apply the fix targeting 14.0.0.

  3. added this to the 13.0.0 milestone on Jul 28, 2023
  4. raulcd commented on Jul 31, 2023

    @raulcd
    Member

    @benibus any news here? I plan to create the new RC tomorrow unless there's possibility for this to be solved soon.

  5. modified the milestones: 13.0.0, 14.0.0 on Aug 1, 2023
  6. benibus commented on Aug 1, 2023

    @benibus
    Contributor

    Should have a fix for this by the end of the week, provided my diagnosis is actually complete.

    At the very least, the PR introduced substantial overhead for calling FieldPath::Get on record batches specifically. It seems this is due to a few instances where usages of RecordBatch::column_data() were replaced with columns(), which is far less trivial (it loads every column atomically on each invocation).

    That being said, the two aren't exactly interchangeable in the implementation so there's some restructuring in order.

  7. added a commit that references this issue on Aug 8, 2023
  8. modified the milestones: 13.0.0, 14.0.0 on Aug 8, 2023
  9. modified the milestones: 14.0.0, 13.0.0 on Aug 8, 2023
  10. added a commit that references this issue on Aug 9, 2023
  11. added a commit that references this issue on Nov 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions