dataclasses.asdict breaks with defaultdict fields #79721

Activity

1. wrmsr commented on Dec 19, 2018

   

   wrmsrmannequin

   on Dec 19, 2018

   MannequinAuthor

   More actions

   _asdict_inner attempts to manually recursively deepcopy dicts by calling type(obj) with a generator of transformed keyvalue tuples @

   cpython/Lib/dataclasses.py

   Line 1080 in b2f642c

   elif isinstance(obj, dict):

   . defaultdicts are dicts so this runs but unlike other dicts their first arg has to be a callable or None:

       import collections
       import dataclasses as dc
   
       @dc.dataclass()
       class C:
           d: dict
   
       c = C(collections.defaultdict(lambda: 3, {}))
       d = dc.asdict(c)

   assert isinstance(d['d'], collections.defaultdict)
   assert d['d']['a'] == 3
   

   =>

       Traceback (most recent call last):
         File "boom.py", line 9, in <module>
           d = dc.asdict(c)
         File "/Users/spinlock/.pyenv/versions/3.7.1/lib/python3.7/dataclasses.py", line 1019, in asdict
           return _asdict_inner(obj, dict_factory)
         File "/Users/spinlock/.pyenv/versions/3.7.1/lib/python3.7/dataclasses.py", line 1026, in _asdict_inner
           value = _asdict_inner(getattr(obj, f.name), dict_factory)
         File "/Users/spinlock/.pyenv/versions/3.7.1/lib/python3.7/dataclasses.py", line 1058, in _asdict_inner
           for k, v in obj.items())
       TypeError: first argument must be callable or None

   I understand that it isn't this bit of code's job to support every dict (and list etc.) subclass under the sun but given defaultdict is stdlib it's imo worth supporting explicitly.
2. wrmsr

   added

   3.7 (EOL)end of lifeend of life

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on Dec 19, 2018
3. ericvsmith

   self-assigned this

   on Dec 19, 2018
4. remilapeyre commented on Dec 30, 2018

   

   remilapeyremannequin

   on Dec 30, 2018

   Mannequin

   More actions

   Hi @wrmsr, this happens because the constructor for collections.defaultdict differs from the one of dict.

   I think the argument that collections.defaultdict is in the stdlib and should be supported is right, the changes in PR bpo-11361 should do what you want.
5. pganssle commented on Sep 24, 2019

   

   pganssle

   on Sep 24, 2019

   Member

   More actions

   Considering that namedtuple is special-cased, I think it's reasonable to special-case defaultdict as well, though it may be worth considering more general solutions that will also work for things other than the standard library. One would be to solve this the same way that other "subclasses may have a different constructor" problems are solved (e.g. float, int, formerly datetime) and ignore the subclass (or selectively ignore it if it's a problem), for example changing _asdict_inner to something like this:

   if isinstance(obj, dict):
       new_keys = tuple((_asdict_inner(k, dict_factory),
                         _asdict_inner(v, dict_factory))
                         for k, v in obj.items())

   try:
       return type(obj)(new_keys)
   except Exception:
       return dict(new_keys)
   

   Another more general alternative would be to add a type registry for asdict, either as an additional parameter or with a new transformer class of some sort. I created a quick proof of concept for this in #60560 to see one way it could look.

   In any case I think it's quite unfortunate that we can't easily just support anything that has a deepcopy defined. There may be some crazy solution that involves passing a class with a custom getitem to the memo argument of copy.deepcopy, but if it's even possible (haven't thought about it enough) I'm not sure it's advisable.
6. pganssle commented on Sep 24, 2019

   

   pganssle

   on Sep 24, 2019

   Member

   More actions

   I checked and it appears that attrs handles this by creating all dicts using the default dict_factory (similar to my original suggestion of just using dict instead of the specific type), if I'm reading this right: https://github.com/python-attrs/attrs/blob/master/src/attr/_funcs.py#L102

   Using attr.asdict seems to bear this out, as defaultdict attributes are converted to dict when the dict factory is not specified.

   I think changing the default behavior like that would be a backwards-incompatible change at this point (and one that it's really hard to warn about, unfortunately), but we could still use the "fall back to dict_factory" behavior by trying to construct a type(obj)(...) and in the case of an exception return dict_factory(...).
7. alexcoca commented on Apr 12, 2021

   

   alexcocamannequin

   on Apr 12, 2021

   Mannequin

   More actions

   I was wondering if this issue is still being tracked for resolution? I found the same bug in Python 3.8.
8. ryx2 commented on Oct 28, 2021

   

   ryx2mannequin

   on Oct 28, 2021

   Mannequin

   More actions

   I am seeing this bug with 3.9.7
9. ezio-melotti

   transferred this issue fromon Apr 10, 2022
10. serhiy-storchaka

    added this to Pickle and copy issues 🥒on May 20, 2022
11. ericvsmith commented on Oct 7, 2022

    

    ericvsmith

    on Oct 7, 2022

    Member

    More actions

    Fixed in #32056. Thanks!
12. ericvsmith

    closed this as completedon Oct 7, 2022
13. ericvsmith

    moved this to Done in Pickle and copy issues 🥒on Oct 7, 2022
14. khaeru

    added 2 commits that reference this issue on Dec 3, 2024

    Backport dataclasses.asdict(…) for Python ≤ 3.11

    ...

    efc5ef8

    Backport dataclasses.asdict(…) for Python ≤ 3.11

    ...

    d5d7778
15. GamzeUnlu95

    added a commit that references this issue on Dec 4, 2024

    Backport dataclasses.asdict(…) for Python ≤ 3.11

    ...

    2bfa357