You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[C++][FlightRPC] Buffer handling change in 12.0.1 causing a lot of warnings being printed #36301
Describe the bug, including details regarding any error messages, version, and platform.
Version: 12.0.1
"An input buffer was poorly aligned. This could lead to crashes or "
"poor performance on some hardware. Please ensure that all Acero "
"sources generate aligned buffers, or change the unaligned buffer "
"handling configuration to silence this warning."
When doing a do_get with pyarrow flight clients now, a warning is triggered for every column in every recordbatch with misaligned buffers, causing a lot of warnings to be printed to console when a table is created from the recordbatches. This is due to flight creating misaligned buffers #32276. I believe this was introduced by #35565 in how it handles unaligned buffers.
As there is no way to change this behaviour in pyarrow, the only I've found to sidestep it is to use copy.deepcopy on the recordbatches before making the pyarrow table.
Apologies for not being able to provide a MWE of the bug
changed the title [-]Buffer handling change in 12.0.1 causing a lot of warnings being printed[/-][+][C++][FlightRPC] Buffer handling change in 12.0.1 causing a lot of warnings being printed[/+]on Jun 27, 2023
I think it would be useful to update the warning message, " Please ensure that all Acero sources generate aligned buffers, or change the unaligned buffer handling configuration to silence this warning.", as this is unclear for Python users what they're suppose to change, as acero is not exposed on the pyarrow side. Also, this being an environment variable is not listed in the Python or C++ docs currently
I agree that updating the warning message would be a good idea. It's located here if you want to propose some new wording (I'm not really sure how to describe this without using the word "Acero"). Also, just to be clear, this usage of unaligned buffers was not recently introduced. It is the way that Flight (well, technically, grpc I think) has always worked. That only thing that is new is that we've started checking for it and issuing a warning when we detect it. In most cases (on modern CPUs) this warning is fairly harmless.
Yes it's not the first time I've come across this misaligned buffer issue, its caused a few downstream issues for me before, pola-rs/polars#6315, i do understand its not an arrow issue. The updated docs LGTM, I'll try and get round to looking at the arrow Python/Flight docs to see where it might be good to put a note about this.
Activity
[-]Buffer handling change in 12.0.1 causing a lot of warnings being printed[/-][+][C++][FlightRPC] Buffer handling change in 12.0.1 causing a lot of warnings being printed[/+]benkimpel commentedon Jun 29, 2023
@RichardHaythorn I'm also seeing this issue in our org, but so far only on windows. Is that, by any chance, the OS you're running?
RichardHaythorn commentedon Jun 30, 2023
@benkimpel This problem is cropping up on Ubuntu 22 for me
benkimpel commentedon Jun 30, 2023
Gotcha, thanks. That's good to know that it's not just windows.
kou commentedon Jul 1, 2023
Does
ACERO_ALIGNMENT_HANDLING=ignoreenvironment variable work for you?RichardHaythorn commentedon Jul 3, 2023
Yes that seems to have done it! I didn't realise the new switch case in the commit was using environment variables
kou commentedon Jul 3, 2023
Can we close this?
RichardHaythorn commentedon Jul 3, 2023
I think it would be useful to update the warning message, " Please ensure that all Acero sources generate aligned buffers, or change the unaligned buffer handling configuration to silence this warning.", as this is unclear for Python users what they're suppose to change, as acero is not exposed on the pyarrow side. Also, this being an environment variable is not listed in the Python or C++ docs currently
kou commentedon Jul 3, 2023
OK. Do you want to open a pull request to improve them?
westonpace commentedon Jul 11, 2023
I agree that updating the warning message would be a good idea. It's located here if you want to propose some new wording (I'm not really sure how to describe this without using the word "Acero"). Also, just to be clear, this usage of unaligned buffers was not recently introduced. It is the way that Flight (well, technically, grpc I think) has always worked. That only thing that is new is that we've started checking for it and issuing a warning when we detect it. In most cases (on modern CPUs) this warning is fairly harmless.
westonpace commentedon Jul 11, 2023
#36622 adds documentation to the environment variables page. We can leave this issue open in case someone wants to improve the error message.
RichardHaythorn commentedon Jul 12, 2023
Yes it's not the first time I've come across this misaligned buffer issue, its caused a few downstream issues for me before, pola-rs/polars#6315, i do understand its not an arrow issue. The updated docs LGTM, I'll try and get round to looking at the arrow Python/Flight docs to see where it might be good to put a note about this.
westonpace commentedon Jul 12, 2023
@RichardHaythorn do you think you could provide a quick review of the wording in #36622 ?