From 0594428e91d6008fc406c2cd3787858fe1273474 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 19:43:54 +0000 Subject: [PATCH 01/15] Defer opt-out mechanism for C-extensions. --- pep-0554.rst | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index b8b47cdc0c3..60cab802338 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -594,6 +594,25 @@ way it supports threads and processes. In fact, the module's maintainer, Davin Potts, has indicated this is a reasonable feature request. However, it is outside the narrow scope of this PEP. +C-extension opt-in/opt-out +-------------------------- + +By using the ``PyModuleDef_Slot`` introduced by PEP 489, we could easily +add a mechanism by which C-extension modules could opt out of support +for subinterpreters. Then the import machinery, when operating in +a subinterpreter, would need to check the module for support. It would +raise an ImportError if unsupported. + +Alternately we could support opting in to subinterpreter support. +However, that would probably exclude many more modules (unnecessarily) +than the opt-out approach. + +The scope of adding the ModuleDef slot and fixing up the import +machinery is non-trivial, but could be worth it. It all depends on +how many extension modules break under subinterpreters. Given the +relatively few cases we know of through mod_wsgi, we can leave this +for later. + References ========== From ee03dbecc83416c996cedd0faf0fa095fcfbd971 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 13 Sep 2017 21:17:09 +0000 Subject: [PATCH 02/15] list() -> list_all(). --- pep-0554.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0554.rst b/pep-0554.rst index 60cab802338..9a227d5f943 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -35,7 +35,7 @@ and use cases. The module provides the following functions: -``list()``:: +``list_all()``:: Return a list of all existing interpreters. From 0f357c7a2c55517c513dd96e132f6a584843f5a9 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 13 Sep 2017 21:17:55 +0000 Subject: [PATCH 03/15] Switch add_recv_fifo() and add_send_fifo(). --- pep-0554.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index 9a227d5f943..562d349df3f 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -127,28 +127,28 @@ The module also provides the following classes: Create a new FIFO, associate the two ends with the involved interpreters, and return the side associated with the interpreter - in which "add_recv_fifo()" was called. A FIFOReader gets tied to - this interpreter. A FIFOWriter gets tied to the interpreter that + in which "add_recv_fifo()" was called. A FIFOWriter gets tied to + this interpreter. A FIFOReader gets tied to the interpreter that called "add_recv_fifo()". The FIFO's name is set to the provided value. If no name is provided then a dynamically generated one is used. If a FIFO with the given name is already associated with this interpreter - (or with the one in which "add_recv_fifo()" was called) then raise + (or with the one in which "add_send_fifo()" was called) then raise KeyError. add_send_fifo(name=None): Create a new FIFO, associate the two ends with the involved interpreters, and return the side associated with the interpreter - in which "add_recv_fifo()" was called. A FIFOWriter gets tied to - this interpreter. A FIFOReader gets tied to the interpreter that + in which "add_recv_fifo()" was called. A FIFOReader gets tied to + this interpreter. A FIFOWriter gets tied to the interpreter that called "add_recv_fifo()". The FIFO's name is set to the provided value. If no name is provided then a dynamically generated one is used. If a FIFO with the given name is already associated with this interpreter - (or with the one in which "add_send_fifo()" was called) then raise + (or with the one in which "add_recv_fifo()" was called) then raise KeyError. remove_fifo(name): From 3236f51930f8123516d37590cdb3ac444081b91b Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 13 Sep 2017 23:36:47 +0000 Subject: [PATCH 04/15] FIFO -> pipe. --- pep-0554.rst | 249 +++++++++++++++++++++++++++++---------------------- 1 file changed, 141 insertions(+), 108 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index 562d349df3f..bfdcdedd4e1 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -33,6 +33,9 @@ provide a high-level interface to subinterpreters and wrap the low-level ``threading`` module. See the `Examples`_ section for concrete usage and use cases. +API for interpreters +-------------------- + The module provides the following functions: ``list_all()``:: @@ -51,7 +54,8 @@ The module provides the following functions: in any thread and will run in whichever thread calls ``interp.run()``. -The module also provides the following classes: + +The module also provides the following class: ``Interpreter(id)``:: @@ -71,9 +75,13 @@ The module also provides the following classes: This may not be called on an already running interpreter. Doing so results in a RuntimeError. - run(source_str): + run(source_str, /, **shared): - Run the provided Python source code in the interpreter. + Run the provided Python source code in the interpreter. Any + keyword arguments are added to the interpreter's execution + namespace. If any of the values are not supported for sharing + between interpreters then RuntimeError gets raised. Currently + only pipes (see "add_pipe()" below) are supported. This may not be called on an already running interpreter. Doing so results in a RuntimeError. @@ -112,116 +120,104 @@ The module also provides the following classes: Supported code: source text. - get_fifo(name): + add_pipe():: - Return the FIFO object with the given name that is associated - with this interpreter. If no such FIFO exists then raise - KeyError. The FIFO will be either a "FIFOReader" or a - "FIFOWriter", depending on which "add_*_fifo()" was called. + Create a new pipe and return the two ends: (recv, send). These + correspond to a "RecvPipe" and a "SendPipe", respectively. Both + ends of the pipe are supported "shared" objects (i.e. may be + safely shared by different interpreters. Thus they may be passed + as keyword arguments to "Interpreter.run()". - list_fifos(): + The current interpreter (where "add_pipe()" was called) and the + target (on which "add_pipe()" was called) are associated with the + pipe. No other interpreters may use the pipe. - Return a list of all fifos associated with the interpreter. + Either end of the pipe must be closed before it will be destroyed. + Once the pipe is closed it will no longer be tied to either + interpreter. - add_recv_fifo(name=None): + list_pipes(): - Create a new FIFO, associate the two ends with the involved - interpreters, and return the side associated with the interpreter - in which "add_recv_fifo()" was called. A FIFOWriter gets tied to - this interpreter. A FIFOReader gets tied to the interpreter that - called "add_recv_fifo()". + Return a list of all open pipes associated with the interpreter. - The FIFO's name is set to the provided value. If no name is - provided then a dynamically generated one is used. If a FIFO - with the given name is already associated with this interpreter - (or with the one in which "add_send_fifo()" was called) then raise - KeyError. +API for sharing data +-------------------- - add_send_fifo(name=None): +The mechanism for passing objects between interpreters is through pipes. +They are created calling ``Interpreter.add_pipe()``. - Create a new FIFO, associate the two ends with the involved - interpreters, and return the side associated with the interpreter - in which "add_recv_fifo()" was called. A FIFOReader gets tied to - this interpreter. A FIFOWriter gets tied to the interpreter that - called "add_recv_fifo()". +``RecvPipe(id)``:: - The FIFO's name is set to the provided value. If no name is - provided then a dynamically generated one is used. If a FIFO - with the given name is already associated with this interpreter - (or with the one in which "add_recv_fifo()" was called) then raise - KeyError. + The receiving end of a pipe. An interpreter may use this to receive + objects from another interpreter. At first only bytes and None will + be supported. - remove_fifo(name): + id: - Drop the association between the named FIFO and this interpreter. - If the named FIFO is not found then raise KeyError. + The pipe's unique ID. + __next__(): -``FIFOReader(name)``:: + Return the next object from the pipe. If none have been sent + then block. - The receiving end of a FIFO. An interpreter may use this to receive - objects from another interpreter. At first only bytes and None will - be supported. + recv(): - name: + Return the next object from the pipe. If none have been sent + then wait until the next send. If the pipe has been closed + then EOFError is raised. - The FIFO's name. + recv_nowait(default=None): - __next__(): + Return the next object from the pipe. If none have been sent + then return the default. If the pipe has been closed + then EOFError is raised. - Return the next bytes object from the pipe. If none have been - pushed on then block. + close(): - pop(*, block=True): + Mark the pipe as closed. This will cause subsequent (or current) + send() and recv() calls to raise EOFError. The pipe will be + destroyed (i.e. garbage collected) after it has been explicitly + closed. This is necessary because pipes are managed by the + Python runtime and not by either interpreter. - Return the next bytes object from the pipe. If none have been - pushed on and "block" is True (the default) then block. - Otherwise return None. + This operation is idempotent. Return True if the pipe was still + open and False otherwise. -``FIFOWriter(name)``:: +``SendPipe(id)``:: - The sending end of a FIFO. An interpreter may use this to send + The sending end of a pipe. An interpreter may use this to send objects to another interpreter. At first only bytes and None will be supported. - name: + id: - The FIFO's name. + The pipe's unique ID. - push(object, *, block=True): + send(obj): - Add the object to the FIFO. If "block" is true then block - until the object is popped off. If the FIFO does not support - the object's type then TypeError is raised. + Send the object to the receiving end of the pipe. Wait until + the object is received. If the pipe does not support the object + then TypeError is raised. Currently only bytes and None are + supported. -About FIFOs ------------ + send_nowait(obj): -Subinterpreters are inherently isolated (with caveats explained below), -in contrast to threads. This enables `a different concurrency model -`_ than currently exists in Python. -`Communicating Sequential Processes`_ (CSP) is the prime example. + Send the object to the receiving end of the pipe. If "block" is true then block + until the object is popped off. If the pipe does not support + the object's type then TypeError is raised. -A key component of this approach to concurrency is message passing. So -providing a message/object passing mechanism alongside ``Interpreter`` -is a fundamental requirement. This proposal includes a basic mechanism -upon which more complex machinery may be built. That basic mechanism -draws inspiration from pipes, queues, and CSP's channels. + close(): -The key challenge here is that sharing objects between interpreters -faces complexity due in part to CPython's current memory model. -Furthermore, in this class of concurrency, the ideal is that objects -only exist in one interpreter at a time. However, this is not practical -for Python so we initially constrain supported objects to ``bytes`` and -``None``. There are a number of strategies we may pursue in the future -to expand supported objects and object sharing strategies. + Mark the pipe as closed. This will cause subsequent (or current) + send() and recv() calls to raise EOFError. The pipe will be + destroyed (i.e. garbage collected) after it has been explicitly + closed. This is necessary because pipes are managed by the + Python runtime and not by either interpreter. -Note that the complexity of object sharing increases as subinterpreters -become more isolated, e.g. after GIL removal. So the mechanism for -message passing needs to be carefully considered. Keeping the API -minimal and initially restricting the supported types helps us avoid -further exposing any underlying complexity to Python users. + This operation is idempotent. Return True if the pipe was still + open and False otherwise. Examples @@ -279,26 +275,24 @@ Handling an exception except KeyError: print("got the error from the subinterpreter") -Synchronize using a FIFO +Synchronize using a pipe ------------------------ :: interp = interpreters.create() - writer = interp.add_recv_fifo('spam') + r, s = interp.add_pipe() def run(): interp.run("""if True: - import interpreters - interp = interpreters.get_current() - reader = interp.get_fifo('spam') - reader.pop() + reader.recv() print("during") - """) + """, + reader=r) t = threading.Thread(target=run) print('before') t.start() print('after') - writer.push(None) + s.send(None) Sharing a file descriptor ------------------------- @@ -306,24 +300,23 @@ Sharing a file descriptor :: interp = interpreters.create() - writer = interp.add_recv_fifo('spam') - reader = interp.add_send_fifo('done') + r1, s1 = interp.add_pipe() + r2, s2 = interp.add_pipe() def run(): interp.run("""if True: - import interpreters - interp = interpreters.get_current() - reader = interp.get_fifo('spam') - writer = interp.get_fifo('done') - fd = reader.pop() + fd = int.from_bytes( + reader.recv(), 'big') for line in os.fdopen(fd): print(line) - writer.push(None) - """) + writer.send(None) + """, + reader=r1, writer=s2) t = threading.Thread(target=run) t.start() with open('spamspamspam') as infile: - writer.push(infile.fileno()) - reader.pop() + fd = infile.fileno().to_bytes(1, 'big') + s.send(fd) + r.recv() Passing objects via pickle -------------------------- @@ -331,27 +324,25 @@ Passing objects via pickle :: interp = interpreters.create() - writer = interp.add_recv_fifo('spam') + r, s = interp.add_pipe() interp.run("""if True: import pickle - import interpreters - interp = interpreters.get_current() - reader = interp.get_fifo('spam') - """) + """, + reader=r) def run(): interp.run("""if True: - data = reader.pop() + data = reader.recv() while data is not None: obj = pickle.loads(data) do_something(obj) - data = reader.pop() + data = reader.recv() """) t = threading.Thread(target=run) t.start() for obj in input: data = pickle.dumps(obj) - writer.push(data) - writer.push(None) + s.send(data) + s.send(None) Rationale @@ -432,6 +423,40 @@ which subinterpreters are worth it. About Subinterpreters ===================== +Shared data +----------- + +Subinterpreters are inherently isolated (with caveats explained below), +in contrast to threads. This enables `a different concurrency model +`_ than is currently readily available in Python. +`Communicating Sequential Processes`_ (CSP) is the prime example. + +A key component of this approach to concurrency is message passing. So +providing a message/object passing mechanism alongside ``Interpreter`` +is a fundamental requirement. This proposal includes a basic mechanism +upon which more complex machinery may be built. That basic mechanism +draws inspiration from pipes, queues, and CSP's channels. [fifo]_ + +The key challenge here is that sharing objects between interpreters +faces complexity due in part to CPython's current memory model. +Furthermore, in this class of concurrency, the ideal is that objects +only exist in one interpreter at a time. However, this is not practical +for Python so we initially constrain supported objects to ``bytes`` and +``None``. There are a number of strategies we may pursue in the future +to expand supported objects and object sharing strategies. + +Note that the complexity of object sharing increases as subinterpreters +become more isolated, e.g. after GIL removal. So the mechanism for +message passing needs to be carefully considered. Keeping the API +minimal and initially restricting the supported types helps us avoid +further exposing any underlying complexity to Python users. + +To make this work, the mutable shared state will be managed by the +Python runtime, not by any of the interpreters. Initially we will +support only one type of objects for shared state: the pipes provided +by ``Interpreter.add_pipe()``. Pipes, in turn, will carefully manage +passing objects between interpreters. + Interpreter Isolation --------------------- @@ -564,7 +589,7 @@ The ``threading`` module provides a number of synchronization primitives for coordinating concurrent operations. This is especially necessary due to the shared-state nature of threading. In contrast, subinterpreters do not share state. Data sharing is restricted to -FIFOs, which do away with the need for explicit synchronization. If +pipes, which do away with the need for explicit synchronization. If any sort of opt-in shared state support is added to subinterpreters in the future, that same effort can introduce synchronization primitives to meet that need. @@ -626,6 +651,14 @@ References https://en.wikipedia.org/wiki/Communicating_sequential_processes https://github.com/futurecore/python-csp +.. [fifo] + https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Pipe + https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue + https://docs.python.org/3/library/queue.html#module-queue + http://stackless.readthedocs.io/en/2.7-slp/library/stackless/channels.html + https://golang.org/doc/effective_go.html#sharing + http://www.jtolds.com/writing/2016/03/go-channels-are-bad-and-you-should-feel-bad/ + .. [caveats] https://docs.python.org/3/c-api/init.html#bugs-and-caveats From 07e203df1797c98fdf2cb8c2a5c67b139243bdfb Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Thu, 14 Sep 2017 00:22:39 +0000 Subject: [PATCH 05/15] pipe -> channel. --- pep-0554.rst | 159 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 95 insertions(+), 64 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index bfdcdedd4e1..b22e95adbb3 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -81,7 +81,7 @@ The module also provides the following class: keyword arguments are added to the interpreter's execution namespace. If any of the values are not supported for sharing between interpreters then RuntimeError gets raised. Currently - only pipes (see "add_pipe()" below) are supported. + only channels (see "create_channel()" below) are supported. This may not be called on an already running interpreter. Doing so results in a RuntimeError. @@ -120,104 +120,131 @@ The module also provides the following class: Supported code: source text. - add_pipe():: +API for sharing data +-------------------- - Create a new pipe and return the two ends: (recv, send). These - correspond to a "RecvPipe" and a "SendPipe", respectively. Both - ends of the pipe are supported "shared" objects (i.e. may be - safely shared by different interpreters. Thus they may be passed - as keyword arguments to "Interpreter.run()". +The mechanism for passing objects between interpreters is through +channels. A channel is a simplex FIFO similar to a pipe. The main +difference is that channels can be associated with zero or more +interpreters on either end. Unlike queues, which are also many-to-many, +channels have no buffer. - The current interpreter (where "add_pipe()" was called) and the - target (on which "add_pipe()" was called) are associated with the - pipe. No other interpreters may use the pipe. +``create_channel()``:: - Either end of the pipe must be closed before it will be destroyed. - Once the pipe is closed it will no longer be tied to either - interpreter. + Create a new channel and return (recv, send), the RecvChannel and + SendChannel corresponding to the ends of the channel. The channel + is not closed and destroyed (i.e. garbage-collected) until the number + of associated interpreters returns to 0. - list_pipes(): + An interpreter gets associated with a channel by calling its "send()" + or "recv()" method. That association gets dropped by calling + "close()" on the channel. - Return a list of all open pipes associated with the interpreter. + Both ends of the channel are supported "shared" objects (i.e. may be + safely shared by different interpreters. Thus they may be passed as + keyword arguments to "Interpreter.run()". -API for sharing data --------------------- +``list_all_channels()``:: -The mechanism for passing objects between interpreters is through pipes. -They are created calling ``Interpreter.add_pipe()``. + Return a list of all open (RecvChannel, SendChannel) pairs. -``RecvPipe(id)``:: - The receiving end of a pipe. An interpreter may use this to receive - objects from another interpreter. At first only bytes and None will - be supported. +``RecvChannel(id)``:: + + The receiving end of a channel. An interpreter may use this to + receive objects from another interpreter. At first only bytes and + None will be supported. id: - The pipe's unique ID. + The channel's unique ID. + + interpreters: + + The list of associated interpreters (those that have called + the "recv()" method). __next__(): - Return the next object from the pipe. If none have been sent - then block. + Return the next object from the channel. If none have been sent + then wait until the next send. recv(): - Return the next object from the pipe. If none have been sent - then wait until the next send. If the pipe has been closed + Return the next object from the channel. If none have been sent + then wait until the next send. If the channel has been closed then EOFError is raised. recv_nowait(default=None): - Return the next object from the pipe. If none have been sent - then return the default. If the pipe has been closed + Return the next object from the channel. If none have been sent + then return the default. If the channel has been closed then EOFError is raised. close(): - Mark the pipe as closed. This will cause subsequent (or current) - send() and recv() calls to raise EOFError. The pipe will be - destroyed (i.e. garbage collected) after it has been explicitly - closed. This is necessary because pipes are managed by the - Python runtime and not by either interpreter. + No longer associate the current interpreter with the channel (on + the receiving end). This is a noop if the interpreter isn't + already associated. - This operation is idempotent. Return True if the pipe was still - open and False otherwise. + Once number of associated interpreters on both ends drops to 0, + the channel is actually marked as closed. At that point any + subsequent (or current) send() and recv() calls will raise + EOFError. The Python runtime will garbage collect all closed + channels. Note that "close()" is automatically called when it + is no longer used in the current interpreter. + This operation is idempotent. Return True if the current + interpreter was still associated with the receiving end of the + channel and False otherwise. -``SendPipe(id)``:: - The sending end of a pipe. An interpreter may use this to send +``SendChannel(id)``:: + + The sending end of a channel. An interpreter may use this to send objects to another interpreter. At first only bytes and None will be supported. id: - The pipe's unique ID. + The channel's unique ID. + + interpreters: + + The list of associated interpreters (those that have called + the "send()" method). send(obj): - Send the object to the receiving end of the pipe. Wait until - the object is received. If the pipe does not support the object - then TypeError is raised. Currently only bytes and None are - supported. + Send the object to the receiving end of the channel. Wait until + the object is received. If the channel does not support the + object then TypeError is raised. Currently only bytes and None + are supported. If the channel has been closed then EOFError is + raised. send_nowait(obj): - Send the object to the receiving end of the pipe. If "block" is true then block - until the object is popped off. If the pipe does not support - the object's type then TypeError is raised. + Send the object to the receiving end of the channel. If the + object is received then return True. Otherwise return False. + If the channel does not support the object then TypeError is + raised. If the channel has been closed then EOFError is raised. close(): - Mark the pipe as closed. This will cause subsequent (or current) - send() and recv() calls to raise EOFError. The pipe will be - destroyed (i.e. garbage collected) after it has been explicitly - closed. This is necessary because pipes are managed by the - Python runtime and not by either interpreter. + No longer associate the current interpreter with the channel (on + the sending end). This is a noop if the interpreter isn't already + associated. + + Once number of associated interpreters on both ends drops to 0, + the channel is actually marked as closed. At that point any + subsequent (or current) send() and recv() calls will raise + EOFError. The Python runtime will garbage collect all closed + channels. Note that "close()" is automatically called when it + is no longer used in the current interpreter. - This operation is idempotent. Return True if the pipe was still - open and False otherwise. + This operation is idempotent. Return True if the current + interpreter was still associated with the sending end of the + channel and False otherwise. Examples @@ -275,17 +302,18 @@ Handling an exception except KeyError: print("got the error from the subinterpreter") -Synchronize using a pipe ------------------------- +Synchronize using a channel +--------------------------- :: interp = interpreters.create() - r, s = interp.add_pipe() + r, s = interpreters.create_channel() def run(): interp.run("""if True: reader.recv() print("during") + reader.close() """, reader=r) t = threading.Thread(target=run) @@ -293,6 +321,7 @@ Synchronize using a pipe t.start() print('after') s.send(None) + s.close() Sharing a file descriptor ------------------------- @@ -300,8 +329,8 @@ Sharing a file descriptor :: interp = interpreters.create() - r1, s1 = interp.add_pipe() - r2, s2 = interp.add_pipe() + r1, s1 = interpreters.create_channel() + r2, s2 = interpreters.create_channel() def run(): interp.run("""if True: fd = int.from_bytes( @@ -324,7 +353,7 @@ Passing objects via pickle :: interp = interpreters.create() - r, s = interp.add_pipe() + r, s = interpreters.create_channel() interp.run("""if True: import pickle """, @@ -336,7 +365,9 @@ Passing objects via pickle obj = pickle.loads(data) do_something(obj) data = reader.recv() - """) + reader.close() + """, + reader=r) t = threading.Thread(target=run) t.start() for obj in input: @@ -453,8 +484,8 @@ further exposing any underlying complexity to Python users. To make this work, the mutable shared state will be managed by the Python runtime, not by any of the interpreters. Initially we will -support only one type of objects for shared state: the pipes provided -by ``Interpreter.add_pipe()``. Pipes, in turn, will carefully manage +support only one type of objects for shared state: the channels provided +by ``create_channel()``. Channels, in turn, will carefully manage passing objects between interpreters. Interpreter Isolation @@ -589,7 +620,7 @@ The ``threading`` module provides a number of synchronization primitives for coordinating concurrent operations. This is especially necessary due to the shared-state nature of threading. In contrast, subinterpreters do not share state. Data sharing is restricted to -pipes, which do away with the need for explicit synchronization. If +channels, which do away with the need for explicit synchronization. If any sort of opt-in shared state support is added to subinterpreters in the future, that same effort can introduce synchronization primitives to meet that need. From c3bce9b571728ac1f2f57c257a78c6ed223acc7a Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Thu, 14 Sep 2017 00:28:55 +0000 Subject: [PATCH 06/15] Add a note about poisoning channels. --- pep-0554.rst | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index b22e95adbb3..be695e09891 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -669,6 +669,19 @@ how many extension modules break under subinterpreters. Given the relatively few cases we know of through mod_wsgi, we can leave this for later. +Poisoning channels +------------------ + +CSP has the concept of poisoning a channel. Once a channel has been +poisoned, and ``send()`` or ``recv()`` call on it will raise a special +exception, effectively ending execution in the interpreter that tried +to use the poisoned channel. + +This could be accomplished by adding a ``poison()`` method to both ends +of the channel. The ``close()`` method could work if it had a ``force`` +option to force the channel closed. Regardless, these semantics are +relatively specialized and can wait. + References ========== From 84e0b94625aafab270b8b5b61b7d0202f5267a23 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Thu, 14 Sep 2017 00:34:20 +0000 Subject: [PATCH 07/15] Add a note about sending channels over channels. --- pep-0554.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index be695e09891..10b235aed0e 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -682,6 +682,15 @@ of the channel. The ``close()`` method could work if it had a ``force`` option to force the channel closed. Regardless, these semantics are relatively specialized and can wait. +Sending channels over channels +------------------------------ + +Some advanced usage of subinterpreters could take advantage of the +ability to send channels over channels, in addiction to bytes and None. +Given that channels will already be multi-interpreter safe, supporting +then in ``RecvChannel.recv()`` wouldn't be a big change. However, this +can wait until the basic functionality has been ironed out. + References ========== From a0245289f1d3329fb0c853c006e2339827211cc2 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Thu, 14 Sep 2017 00:52:36 +0000 Subject: [PATCH 08/15] Add some rejected ideas. --- pep-0554.rst | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 10b235aed0e..20bf2946c33 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -692,6 +692,48 @@ then in ``RecvChannel.recv()`` wouldn't be a big change. However, this can wait until the basic functionality has been ironed out. +Rejected Ideas +============== + +Explicit channel association +---------------------------- + +Interpreters are implicitly associated with channels upon ``recv()`` and +``send()`` calls. They are de-associated with ``close()`` calls. The +alternative would be explicit methods. It would be either +``add_channel()`` and ``remove_channel()`` methods on ``Interpreter`` +objects or something similar on channel objects. + +In practice, this level of management shouldn't be necessary for users. +So adding more explicit support would only add clutter to the API. + +Use pipes instead of channels +----------------------------- + +A pipe would be a simplex FIFO between exactly two interpreters. For +most use cases this would be sufficient. It could potentially simplify +the implementation as well. However, it isn't a big step to supporting +a many-to-many simplex FIFO via channels. Also, with pipes the API +ends up being slightly more complicated, requiring naming the pipes. + +Use queues instead of channels +------------------------------ + +The main difference between queues and channels is that queues support +buffering. This would complicate the blocking semantics of ``recv()`` +and ``send()``. Also, queues can be built on top of channels. + +"enumerate" +----------- + +The ``list_all()`` function provides the list of all interpreters. +In the threading module, which partly inspired the proposed API, the +function is called ``enumerate()``. The name is different here to +avoid confusing Python users that are not already familiar with the +threading API. For them "enumerate" is rather unclear, whereas +"list_all" is clear. + + References ========== From 159f437d23b539edaf74050a7da8679b474cb0b9 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Thu, 14 Sep 2017 01:01:14 +0000 Subject: [PATCH 09/15] Add a note about resetting __main__. --- pep-0554.rst | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 20bf2946c33..13525401757 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -691,6 +691,27 @@ Given that channels will already be multi-interpreter safe, supporting then in ``RecvChannel.recv()`` wouldn't be a big change. However, this can wait until the basic functionality has been ironed out. +Reseting __main__ +----------------- + +As proposed, every call to ``Interpreter.run()`` will execute in the +namespace of the interpreter's existing ``__main__`` module. This means +that data persists there between ``run()`` calls. Sometimes this isn't +desireable and you want to execute in a fresh ``__main__``. Also, +you don't necessarily want to leak objects there that you aren't using +any more. + +Solutions include: + +* a ``create()`` arg to indicate resetting ``__main__`` after each + ``run`` call +* an ``Interpreter.reset_main`` flag to support opting in or out + after the fact +* an ``Interpreter.reset_main()`` method to opt in when desired + +This isn't a critical feature initially. It can wait until later +if desirable. + Rejected Ideas ============== From 5ce03406ae60dc19f2259fa119edf15470d73886 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Thu, 14 Sep 2017 01:08:52 +0000 Subject: [PATCH 10/15] Add an open question about leaking exceptions. --- pep-0554.rst | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 13525401757..d995ef0628b 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -567,6 +567,23 @@ Alternate Python Implementations TBD +Open Questions +============== + +Leaking exceptions across interpreters +-------------------------------------- + +As currently proposed, uncaught exceptions from ``run()`` propagate +to the frame that called it. However, this means that exception +objects are leaking across the inter-interpreter boundary. Likewise, +the frames in the traceback potentially leak. + +While that might not be a problem currently, it would be a problem once +interpreters get better isolation relative to memory management (which +is necessary to stop sharing the GIL between interpreters). So the +semantics of how the exceptions propagate needs to be resolved. + + Deferred Functionality ====================== From c1b94b9629a041b36184e2a93e8ddd40758e9364 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Thu, 14 Sep 2017 01:11:44 +0000 Subject: [PATCH 11/15] Add a note about ints in channels. --- pep-0554.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index d995ef0628b..25efc87adee 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -729,6 +729,14 @@ Solutions include: This isn't a critical feature initially. It can wait until later if desirable. +Support passing ints in channels +-------------------------------- + +Passing ints around should be fine and ultimately is probably +desirable. However, we can get by with serializing them as bytes +for now. The goal is a minimal API for the sake of basic +functionality at first. + Rejected Ideas ============== From 6d42c942d837717b2303476d68bc21fe8296bd4e Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Thu, 14 Sep 2017 01:18:19 +0000 Subject: [PATCH 12/15] Add an open question about buffers. --- pep-0554.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 25efc87adee..1672a4377e3 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -583,6 +583,18 @@ interpreters get better isolation relative to memory management (which is necessary to stop sharing the GIL between interpreters). So the semantics of how the exceptions propagate needs to be resolved. +Initial support for buffers in channels +--------------------------------------- + +An alternative to support for bytes in channels in support for +read-only buffers (the PEP 3119 kind). Then ``recv()`` would return +a memoryview to expose the buffer in a zero-copy way. This is similar +to what ``multiprocessing.Connection`` supports. [mp-conn] + +Switching to such an approach would help resolve questions of how +passing bytes through channels will work once we isolate memory +management in interpreters. + Deferred Functionality ====================== @@ -820,6 +832,9 @@ References .. [global-atexit] https://bugs.python.org/issue6531 +.. [mp-conn] + https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Connection + Copyright ========= From 2986ff36ee9ec01f953964bebf22cf4540d8d2eb Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Thu, 14 Sep 2017 01:23:26 +0000 Subject: [PATCH 13/15] Add a note about FDs and sockets. --- pep-0554.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 1672a4377e3..a747624f116 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -749,6 +749,15 @@ desirable. However, we can get by with serializing them as bytes for now. The goal is a minimal API for the sake of basic functionality at first. +File descriptors and sockets in channels +---------------------------------------- + +Given that file descriptors and sockets are process-global resources, +support for passing them through channels is a reasonable idea. They +would be a good candidate for the first effort at expanding the types +that channels support. They aren't strictly necessary for the initial +API. + Rejected Ideas ============== From 2001d3d2fbee50421b534f0dbccc4db962763264 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Thu, 14 Sep 2017 01:27:16 +0000 Subject: [PATCH 14/15] Fix the description of close(). --- pep-0554.rst | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index a747624f116..d0492cb7efc 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -185,14 +185,15 @@ channels have no buffer. No longer associate the current interpreter with the channel (on the receiving end). This is a noop if the interpreter isn't - already associated. + already associated. Once an interpreter is no longer associated + with the channel, subsequent (or current) send() and recv() calls + from that interpreter will raise EOFError. Once number of associated interpreters on both ends drops to 0, - the channel is actually marked as closed. At that point any - subsequent (or current) send() and recv() calls will raise - EOFError. The Python runtime will garbage collect all closed - channels. Note that "close()" is automatically called when it - is no longer used in the current interpreter. + the channel is actually marked as closed. The Python runtime + will garbage collect all closed channels. Note that "close()" is + automatically called when it is no longer used in the current + interpreter. This operation is idempotent. Return True if the current interpreter was still associated with the receiving end of the @@ -233,14 +234,15 @@ channels have no buffer. No longer associate the current interpreter with the channel (on the sending end). This is a noop if the interpreter isn't already - associated. + associated. Once an interpreter is no longer associated with the + channel, subsequent (or current) send() and recv() calls from that + interpreter will raise EOFError. Once number of associated interpreters on both ends drops to 0, - the channel is actually marked as closed. At that point any - subsequent (or current) send() and recv() calls will raise - EOFError. The Python runtime will garbage collect all closed - channels. Note that "close()" is automatically called when it - is no longer used in the current interpreter. + the channel is actually marked as closed. The Python runtime + will garbage collect all closed channels. Note that "close()" is + automatically called when it is no longer used in the current + interpreter. This operation is idempotent. Return True if the current interpreter was still associated with the sending end of the From 803a88e63809baa65f9f1a9b8899e788e89305fe Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Thu, 14 Sep 2017 01:30:15 +0000 Subject: [PATCH 15/15] Drop support for None in channels. --- pep-0554.rst | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index d0492cb7efc..f6f7992cb11 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -152,8 +152,8 @@ channels have no buffer. ``RecvChannel(id)``:: The receiving end of a channel. An interpreter may use this to - receive objects from another interpreter. At first only bytes and - None will be supported. + receive objects from another interpreter. At first only bytes will + be supported. id: @@ -203,8 +203,8 @@ channels have no buffer. ``SendChannel(id)``:: The sending end of a channel. An interpreter may use this to send - objects to another interpreter. At first only bytes and None will - be supported. + objects to another interpreter. At first only bytes will be + supported. id: @@ -219,8 +219,8 @@ channels have no buffer. Send the object to the receiving end of the channel. Wait until the object is received. If the channel does not support the - object then TypeError is raised. Currently only bytes and None - are supported. If the channel has been closed then EOFError is + object then TypeError is raised. Currently only bytes are + supported. If the channel has been closed then EOFError is raised. send_nowait(obj): @@ -322,7 +322,7 @@ Synchronize using a channel print('before') t.start() print('after') - s.send(None) + s.send(b'') s.close() Sharing a file descriptor @@ -339,7 +339,7 @@ Sharing a file descriptor reader.recv(), 'big') for line in os.fdopen(fd): print(line) - writer.send(None) + writer.send(b'') """, reader=r1, writer=s2) t = threading.Thread(target=run) @@ -363,7 +363,7 @@ Passing objects via pickle def run(): interp.run("""if True: data = reader.recv() - while data is not None: + while data: obj = pickle.loads(data) do_something(obj) data = reader.recv() @@ -375,7 +375,7 @@ Passing objects via pickle for obj in input: data = pickle.dumps(obj) s.send(data) - s.send(None) + s.send(b'') Rationale @@ -474,9 +474,9 @@ The key challenge here is that sharing objects between interpreters faces complexity due in part to CPython's current memory model. Furthermore, in this class of concurrency, the ideal is that objects only exist in one interpreter at a time. However, this is not practical -for Python so we initially constrain supported objects to ``bytes`` and -``None``. There are a number of strategies we may pursue in the future -to expand supported objects and object sharing strategies. +for Python so we initially constrain supported objects to ``bytes``. +There are a number of strategies we may pursue in the future to expand +supported objects and object sharing strategies. Note that the complexity of object sharing increases as subinterpreters become more isolated, e.g. after GIL removal. So the mechanism for @@ -717,10 +717,10 @@ Sending channels over channels ------------------------------ Some advanced usage of subinterpreters could take advantage of the -ability to send channels over channels, in addiction to bytes and None. -Given that channels will already be multi-interpreter safe, supporting -then in ``RecvChannel.recv()`` wouldn't be a big change. However, this -can wait until the basic functionality has been ironed out. +ability to send channels over channels, in addition to bytes. Given +that channels will already be multi-interpreter safe, supporting then +in ``RecvChannel.recv()`` wouldn't be a big change. However, this can +wait until the basic functionality has been ironed out. Reseting __main__ -----------------