Skip to content

[Python] Remove deprecated pyarrow.serialization functionality #29705

Description

@asfimport

Reporter: Joris Van den Bossche / @jorisvandenbossche

Note: This issue was originally created as ARROW-14115. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Sep 24, 2021

    @asfimport
    Author

    Joris Van den Bossche / @jorisvandenbossche:
    @pitrou I was looking into this what removing it would involve. But so we still have one place internally where we use the (de)serialization machinery ourselves, i.e. in plasma.

    In the Python plasma code, we use it for the PlasmaClient.put/PlasmaClient.get to put/get generic Python objects in the plasma store (

    serialized = pyarrow.lib._serialize(value, serialization_context)
    buffer = self.create(target_id, serialized.total_bytes)
    stream = pyarrow.FixedSizeBufferWriter(buffer)
    stream.set_memcopy_threads(memcopy_threads)
    serialized.write_to(stream)
    ). This is something we could replace with storing the pickled object in the store? (so using pickling instead of our own custom (de)serialization)

    In addition, we also use part of the arrow::py serialize functionality in the plasma C++ code: WriteNdarrayHeader (

    ARROW_CHECK_OK(arrow::py::WriteNdarrayHeader(arrow_dtype, shape, 0, &mock));
    ) and NdarrayFromBuffer (
    ARROW_CHECK_OK(arrow::py::NdarrayFromBuffer(object_buffer.data, &ndarray));
    ). It is relatively easy to only keep those two functions in (de)serialize.cc and remove the rest we no longer need. But are you aware of other internal utilities that might cover that as well?

  2. asfimport commented on Sep 27, 2021

    @asfimport
    Author

    Antoine Pitrou / @pitrou:
    Isn't Plasma explicitly deprecated or did we forget to raise any warnings?

  3. asfimport commented on Sep 28, 2021

    @asfimport
    Author

    Joris Van den Bossche / @jorisvandenbossche:
    No, I don't think we ever officially / explicitly decided to deprecate Plasma (although there are regularly mailing list threads asking about plasma status confirming it is basically unmaintained)

  4. added this to the 11.0.0 milestone on Jan 11, 2023
  5. removed this from the 11.0.0 milestone on Jan 11, 2023
  6. added this to the 12.0.0 milestone on Apr 6, 2023
  7. added a commit that references this issue on Apr 6, 2023
  8. added a commit that references this issue on May 11, 2023
  9. added a commit that references this issue on May 15, 2023
  10. added 2 commits that reference this issue on Jun 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions