Skip to content

Opt out serialization/deserialization for heap type #85224

Closed
Closed
Opt out serialization/deserialization for heap type#85224
Labels
3.10 (EOL)end of life3.9 (EOL)end of lifetopic-C-APItype-bugAn unexpected behavior, bug, or error

Description

@corona10
corona10
opened on Jun 20, 2020
BPO 41052
Nosy @pitrou, @vstinner, @serhiy-storchaka, @corona10, @shihai1991
PRs
  • bpo-41052: Opt out serialization/deserialization for _random.Random #21002
  • [3.9] bpo-41052: Opt out serialization/deserialization for _random.Random (GH-21002). #21030
  • bpo-41052: Optout serialization/deserialization for blake2s/b #22189
  • bpo-41052: Fix pickling heap types implemented in C with protocols 0 and 1 #22870
  • [3.9] bpo-41052: Fix pickling heap types implemented in C with protocols 0 and 1 (GH-22870). #22963
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-10-25.16:09:30.699>
created_at = <Date 2020-06-20.13:17:29.949>
labels = ['expert-C-API', 'type-bug', '3.9', '3.10']
title = 'Opt out serialization/deserialization for heap type'
updated_at = <Date 2020-10-27.04:22:53.436>
user = 'https://github.com/corona10'

    bugs.python.org fields:

    activity = <Date 2020-10-27.04:22:53.436>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-25.16:09:30.699>
closer = 'serhiy.storchaka'
components = ['C API']
creation = <Date 2020-06-20.13:17:29.949>
creator = 'corona10'
dependencies = []
files = []
hgrepos = []
issue_num = 41052
keywords = ['patch']
message_count = 10.0
messages = ['371937', '371992', '371993', '376951', '379214', '379223', '379251', '379532', '379585', '379735']
nosy_count = 5.0
nosy_names = ['pitrou', 'vstinner', 'serhiy.storchaka', 'corona10', 'shihai1991']
pr_nums = ['21002', '21030', '22189', '22870', '22963']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41052'
versions = ['Python 3.9', 'Python 3.10']

    Activity

    1. corona10 commented on Jun 20, 2020

      @corona10
      corona10
      on Jun 20, 2020
      MemberAuthor

      See https://bugs.python.org/issue40077#msg371813

      We noticed that heap type has different behavior about serialization/deserialization.

      Basically it can occur the regression issues.

      Two things needed.

      1. opt out serialization/deserialization for converted modules.
      2. Add unit tests to check whether their serialization is blocked.
      3. If the module is already ported to 3.9 backport patch is needed.
        Long term
      • Add the object.reduce() and/or update pickle can be smarter
    2. corona10
      added
      3.9 (EOL)end of life
      3.10 (EOL)end of life
      topic-C-API
      type-bugAn unexpected behavior, bug, or error
      on Jun 20, 2020

    3. corona10 commented on Jun 21, 2020

      @corona10
      corona10
      on Jun 21, 2020
      MemberAuthor

      New changeset 6989af0 by Dong-hee Na in branch 'master':
      bpo-41052: Opt out serialization/deserialization for _random.Random (GH-21002)
      6989af0

    4. corona10 commented on Jun 21, 2020

      @corona10
      corona10
      on Jun 21, 2020
      MemberAuthor

      New changeset 814b07b by Dong-hee Na in branch '3.9':
      [3.9] bpo-41052: Opt out serialization/deserialization for _random.Random (GH-21002). (GH-21030)
      814b07b

    5. serhiy-storchaka commented on Sep 15, 2020

      @serhiy-storchaka
      serhiy-storchaka
      on Sep 15, 2020
      Member

      There are other heap types implemented in C in the stdlib and third-party libraries for which pickling with protocols 0 and 1 works incorrectly. It would be better to fix copyreg._reduce_ex() instead of disabling pickling for these types one by one.

      From PEP-307:

      Let D be the class on the object to be pickled. First, find the nearest base class that is implemented in C (either as a built-in type or as a type defined by an extension class). Call this base class B, and the class of the object to be pickled D. Unless B is the class 'object', instances of class B must be picklable, either by having built-in support (as defined in the above three bullet points), or by having a non-default __reduce__ implementation. B must not be the same class as D (if it were, it would mean that D is not implemented in Python).

      The problem is with determining which class is implemented in C. The current code implies that heap types are implemented in Python, and static types are implemented in C. It is not always true, because some heap types can be implemented in C.

    6. corona10 commented on Oct 21, 2020

      @corona10
      corona10
      on Oct 21, 2020
      MemberAuthor

      It would be better to fix copyreg._reduce_ex() instead of disabling pickling for these types one by one

      I agree :)

    7. pitrou commented on Oct 21, 2020

      @pitrou
      pitrou
      on Oct 21, 2020
      Member

      At some point, pickle protocols 0 and 1 should be deprecated and slated for removal. Protocol 2 is 17 years ago already, and protocol 3 has become the default in Python 3.

      That would probably be a better use of contributor time than trying to make heap types compatible with those obsolete protocols.

    8. serhiy-storchaka commented on Oct 21, 2020

      @serhiy-storchaka
      serhiy-storchaka
      on Oct 21, 2020
      Member

      I came to the same conclusion after trying to fix it. But we cannot just do it now. We have to fix bugs in Python 3.9 and support protocols 0 and 1 for some deprecation period.

      Also protocols 0 and 1 are used in some third-party implementations for other programming languages, so they can be used for interoperability with other languages.

      The proposed patch adds additional check. It uses the fact that the __new__ attribute for classes implemented in C is a bultin function with __self__ pointing to this class (it is created in tp_new_wrapper()).

    9. serhiy-storchaka commented on Oct 24, 2020

      @serhiy-storchaka
      serhiy-storchaka
      on Oct 24, 2020
      Member

      New changeset 8cd1dba by Serhiy Storchaka in branch 'master':
      bpo-41052: Fix pickling heap types implemented in C with protocols 0 and 1 (GH-22870)
      8cd1dba

    10. serhiy-storchaka commented on Oct 25, 2020

      @serhiy-storchaka
      serhiy-storchaka
      on Oct 25, 2020
      Member

      New changeset 0aaecb3 by Serhiy Storchaka in branch '3.9':
      [3.9] bpo-41052: Fix pickling heap types implemented in C with protocols 0 and 1 (GH-22870). (GH-22963)
      0aaecb3

    11. serhiy-storchaka
      closed this as completedon Oct 25, 2020
    12. serhiy-storchaka
      closed this as completedon Oct 25, 2020

    13. vstinner commented on Oct 27, 2020

      @vstinner
      vstinner
      on Oct 27, 2020
      Member

      Thanks for the fix Serhiy!

    14. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of life3.9 (EOL)end of lifetopic-C-APItype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions