Skip to content

[C++][FlightRPC] Buffer handling change in 12.0.1 causing a lot of warnings being printed #36301

Open

Description

@RichardHaythorn

Describe the bug, including details regarding any error messages, version, and platform.

Version: 12.0.1

"An input buffer was poorly aligned. This could lead to crashes or "
"poor performance on some hardware. Please ensure that all Acero "
"sources generate aligned buffers, or change the unaligned buffer "
"handling configuration to silence this warning."

When doing a do_get with pyarrow flight clients now, a warning is triggered for every column in every recordbatch with misaligned buffers, causing a lot of warnings to be printed to console when a table is created from the recordbatches. This is due to flight creating misaligned buffers #32276. I believe this was introduced by #35565 in how it handles unaligned buffers.

As there is no way to change this behaviour in pyarrow, the only I've found to sidestep it is to use copy.deepcopy on the recordbatches before making the pyarrow table.

Apologies for not being able to provide a MWE of the bug

Component(s)

C++, Python

Activity

  1. changed the title [-]Buffer handling change in 12.0.1 causing a lot of warnings being printed[/-] [+][C++][FlightRPC] Buffer handling change in 12.0.1 causing a lot of warnings being printed[/+] on Jun 27, 2023
  2. benkimpel commented on Jun 29, 2023

    @benkimpel

    @RichardHaythorn I'm also seeing this issue in our org, but so far only on windows. Is that, by any chance, the OS you're running?

  3. RichardHaythorn commented on Jun 30, 2023

    @RichardHaythorn
    Author

    @benkimpel This problem is cropping up on Ubuntu 22 for me

  4. benkimpel commented on Jun 30, 2023

    @benkimpel

    Gotcha, thanks. That's good to know that it's not just windows.

  5. kou commented on Jul 1, 2023

    @kou
    Member

    Does ACERO_ALIGNMENT_HANDLING=ignore environment variable work for you?

  6. RichardHaythorn commented on Jul 3, 2023

    @RichardHaythorn
    Author

    Yes that seems to have done it! I didn't realise the new switch case in the commit was using environment variables

  7. kou commented on Jul 3, 2023

    @kou
    Member

    Can we close this?

  8. RichardHaythorn commented on Jul 3, 2023

    @RichardHaythorn
    Author

    I think it would be useful to update the warning message, " Please ensure that all Acero sources generate aligned buffers, or change the unaligned buffer handling configuration to silence this warning.", as this is unclear for Python users what they're suppose to change, as acero is not exposed on the pyarrow side. Also, this being an environment variable is not listed in the Python or C++ docs currently

  9. kou commented on Jul 3, 2023

    @kou
    Member

    OK. Do you want to open a pull request to improve them?

  10. westonpace commented on Jul 11, 2023

    @westonpace
    Member

    I agree that updating the warning message would be a good idea. It's located here if you want to propose some new wording (I'm not really sure how to describe this without using the word "Acero"). Also, just to be clear, this usage of unaligned buffers was not recently introduced. It is the way that Flight (well, technically, grpc I think) has always worked. That only thing that is new is that we've started checking for it and issuing a warning when we detect it. In most cases (on modern CPUs) this warning is fairly harmless.

  11. westonpace commented on Jul 11, 2023

    @westonpace
    Member

    #36622 adds documentation to the environment variables page. We can leave this issue open in case someone wants to improve the error message.

  12. RichardHaythorn commented on Jul 12, 2023

    @RichardHaythorn
    Author

    Yes it's not the first time I've come across this misaligned buffer issue, its caused a few downstream issues for me before, pola-rs/polars#6315, i do understand its not an arrow issue. The updated docs LGTM, I'll try and get round to looking at the arrow Python/Flight docs to see where it might be good to put a note about this.

  13. westonpace commented on Jul 12, 2023

    @westonpace
    Member

    @RichardHaythorn do you think you could provide a quick review of the wording in #36622 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions