Update multiprocessing.managers.ListProxy and multiprocessing.managers.DictProxy

Feature or enhancement

Standard lists accept clear as a shortcut for del xs[:]

xs = ['a']
xs.clear()

However, multiprocessing.ListProxy omitted support for clear().

from multiprocessing import Manager

with Manager() as manager:
    xs = manager.list()
    xs.clear()

raises the following exception

AttributeError: 'ListProxy' object has no attribute 'clear'

Pitch

If clear is supported in a standard list, it should be supported in a multiprocessing list!

Previous discussion

This issue was not previously discussed.

Linked PRs

gh-103134: Update multiprocessing.managers.ListProxy and multiprocessing.managers.DictProxy #103133

ListProxy was written before list.clear and list.copy were added in 3.3. Since manager.list has all the other list methods, it seems straightforward to add the new methods.

Dicts got reversed in 3.8 and | and |= in 3.9. I think we should expand this to updating all proxy types.

Update ListProxy and DictProxy

@benjaminp

Updated the pull request to add xs.copy and d.fromkeys, d | {}, {} | d, reversed(d), d |= {} as you recommended.

Waiting to hear back from @benjaminp to see whether it is safe to replace BaseListProxy with ListProxy and just bring __iadd__ and __imul__ directly as you suggested.

Also, is there a place to put unit tests? I guess to test this properly, we would need to launch two processes and check that changes in one process are reflected in changes in the other process... Not sure how we would write a unit test for that, although we can certainly test it manually.

As an update, I did try removing BaseListProxy and adding __iadd__ and __imul__ directly to ListProxy but it didn't work. I'm not sure why. Here is the code that I used to test this functionality manually.

from multiprocessing import Manager, Process

def f(x):
    x += [1]
    x *= 2
    print(x)

def g(d):
    d |= {'b': 2}
    print(d)

with Manager() as manager:

    x = manager.list()
    x.append(0)
    p = Process(target=f, args=(x,))
    p.start()
    p.join()
    q = Process(target=print, args=(x,))
    q.start()
    q.join()

    d = manager.dict()
    d['a'] = 1
    p = Process(target=g, args=(d,))
    p.start()
    p.join()
    q = Process(target=print, args=(d,))
    q.start()
    q.join()

RESULTS

ACTUAL after replacing BaseListProxy with ListProxy (the __imul__ doesn't seem to work).
ACTUAL when DictProxy is missing __ior__

[0, 1, 0, 1]
[0, 1]
{'a': 1, 'b': 2}
{'a': 1}

EXPECTED and ACTUAL with current commits for this pull request (keep BaseListProxy and bring DictProxy.__ior__) -- works with current pull request

[0, 1, 0, 1]
[0, 1, 0, 1]
{'a': 1, 'b': 2}
{'a': 1, 'b': 2}

RESULTS

ACTUAL after replacing BaseListProxy with ListProxy (the __imul__ doesn't seem to work). ACTUAL when DictProxy is missing __ior__
[0, 1, 0, 1]
[0, 1]
{'a': 1, 'b': 2}
{'a': 1}
EXPECTED and ACTUAL with current commits for this pull request (keep BaseListProxy and bring DictProxy.__ior__) -- works with current pull request
[0, 1, 0, 1]
[0, 1, 0, 1]
{'a': 1, 'b': 2}
{'a': 1, 'b': 2}

Digging for one hour, I found it actually pretty simple.

Try this one out,

def f(x):
    x += [1]
    print(type(x))
    x *= 2

When you find x has type list but not <class 'multiprocessing.managers.ListProxy'>, you know you screwed up.

class ListProxy(BaseListProxy):
    def __iadd__(self, value):
        self._callmethod('extend', (value,))
        return self
    def __imul__(self, value):
        self._callmethod('__imul__', (value,))
        return self

This return self plays key role here. I understand the whole thing after seeing this.

cpython/Lib/multiprocessing/managers.py

Lines 808 to 811 in a0305c5

    
               def _callmethod(self, methodname, args=(), kwds={}): 
        
                   ''' 
        
                   Try to call a method of the referent and return a copy of the result 
        
                   '''

@sunmy2019

Fix __ior__ and add tests thanks to python#103134 (comment) @sunmy2019

@sunmy2019

When you find x has type list but not <class 'multiprocessing.managers.ListProxy'>, you know you screwed up.
def f(x):
    x += [1]
    print(type(x))
    x *= 2

@sunmy2019 thanks so much for the important hint. I was able to make obj |= {'bar': 6} work properly after overriding __ior__ using the same technique used for ListProxy.

BaseDictProxy = MakeProxyType('DictProxy', (
    '__contains__', '__delitem__', '__getitem__', '__ior__', '__iter__',
    '__len__', '__or__', '__reversed__', '__ror__', '__setitem__', 'clear',
    'copy', 'fromkeys', 'get', 'items', 'keys', 'pop', 'popitem',
    'setdefault', 'update', 'values',
    ))
class DictProxy(BaseDictProxy):
    _method_to_typeid_ = {
        '__iter__': 'Iterator',
    }
    def __ior__(self, value):
        self._callmethod('__ior__', (value,))
        return self

I finished adding the tests and the pull request is ready for review.

gh-103134: Update multiprocessing.managers.ListProxy and DictProxy (G…

thanks for taking this on!

pythongh-103134: Update multiprocessing.managers.ListProxy and DictPr…

pythongh-84559: pythongh-103134: Whats new 3.14 entries for multiproc…

gh-84559: gh-103134: Whats new 3.14 entries for multiprocessing. (GH-…

pythongh-84559: pythongh-103134: Whats new 3.14 entries for multiproc…

Navigation Menu

Repository navigation

Update multiprocessing.managers.ListProxy and multiprocessing.managers.DictProxy #103134

Feature or enhancement

Pitch

Previous discussion

Linked PRs

RESULTS

RESULTS

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	def _callmethod(self, methodname, args=(), kwds={}):
	'''
	Try to call a method of the referent and return a copy of the result
	'''

Update multiprocessing.managers.ListProxy and multiprocessing.managers.DictProxy #103134

Description

Feature or enhancement

Pitch

Previous discussion

Linked PRs

Activity

terryjreedy commented on Mar 30, 2023

terryjreedy commented on Mar 30, 2023

invisibleroads commented on Mar 31, 2023

invisibleroads commented on Mar 31, 2023

RESULTS

sunmy2019 commented on Apr 2, 2023

RESULTS

invisibleroads commented on Apr 30, 2023

gpshead commented on May 20, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions