From 1bfc96eae3f27991a2de7bc888ee9c70f05dd21f Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 15 Sep 2017 23:56:25 +0000 Subject: [PATCH 01/26] Update the post history. --- pep-0554.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0554.rst b/pep-0554.rst index f6f7992cb11..1655a0fe603 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -6,7 +6,7 @@ Type: Standards Track Content-Type: text/x-rst Created: 2017-09-05 Python-Version: 3.7 -Post-History: +Post-History: 07-Sep-2017, 08-Sep-2017, 13-Sep-2017 Abstract From 4fbb993614182016f347ee32068c4bfd981c5d85 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Sat, 16 Sep 2017 00:03:55 +0000 Subject: [PATCH 02/26] Add a link about the bug rate. --- pep-0554.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/pep-0554.rst b/pep-0554.rst index 1655a0fe603..587119101fe 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -442,7 +442,8 @@ Consequently, projects that publish extension modules may face an increased maintenance burden as their users start using subinterpreters, where their modules may break. This situation is limited to modules that use C globals (or use libraries that use C globals) to store -internal state. +internal state. For numpy, the reported-bug rate is one every 6 +months. [bug-rate]_ Ultimately this comes down to a question of how often it will be a problem in practice: how many projects would be affected, how often @@ -846,6 +847,9 @@ References .. [mp-conn] https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Connection +.. [bug-rate] + https://mail.python.org/pipermail/python-ideas/2017-September/047094.html + Copyright ========= From 7a6e00fc9ed017ac039305bb52399647232eca72 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Sat, 16 Sep 2017 00:04:07 +0000 Subject: [PATCH 03/26] Add a note about JEP. --- pep-0554.rst | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index 587119101fe..1a25bffde13 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -546,11 +546,12 @@ Existing Usage -------------- Subinterpreters are not a widely used feature. In fact, the only -documented case of wide-spread usage is -`mod_wsgi `_. On the one -hand, this case provides confidence that existing subinterpreter support -is relatively stable. On the other hand, there isn't much of a sample -size from which to judge the utility of the feature. +documented cases of wide-spread usage are +`mod_wsgi `_and +`JEP `_. On the one hand, this case +provides confidence that existing subinterpreter support is relatively +stable. On the other hand, there isn't much of a sample size from which +to judge the utility of the feature. Provisional Status From 84eec8fb565682642ae72d919ce91f5de969ec2f Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 19:21:18 +0000 Subject: [PATCH 04/26] Clarify the semantics of closing channels. --- pep-0554.rst | 81 +++++++++++++++++++++++++--------------------------- 1 file changed, 39 insertions(+), 42 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index 1a25bffde13..fcca7bd9fd9 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -161,43 +161,50 @@ channels have no buffer. interpreters: - The list of associated interpreters (those that have called - the "recv()" method). + The list of associated interpreters: those that have called + the "recv()" or "__next__()" methods and haven't called "close()". __next__(): Return the next object from the channel. If none have been sent - then wait until the next send. + then wait until the next send. See the recv() method. recv(): Return the next object from the channel. If none have been sent - then wait until the next send. If the channel has been closed - then EOFError is raised. + then wait until the next send. This associates the current + interpreter with the channel. + + If the channel is already closed (see the close() method) + then raise EOFError. If the channel isn't closed, but the current + interpreter already called the "close()" method (which drops its + association with the channel) then raise ValueError. recv_nowait(default=None): Return the next object from the channel. If none have been sent - then return the default. If the channel has been closed - then EOFError is raised. + then return the default. Otherwise, this is the same as the + "recv()" method. close(): No longer associate the current interpreter with the channel (on - the receiving end). This is a noop if the interpreter isn't - already associated. Once an interpreter is no longer associated - with the channel, subsequent (or current) send() and recv() calls - from that interpreter will raise EOFError. + the receiving end) and block future association (via the "recv()" + method. If the interpreter was never associated with the channel + then still block future association. Once an interpreter is no + longer associated with the channel, subsequent (or current) send() + and recv() calls from that interpreter will raise ValueError + (or EOFError if the channel is actually marked as closed). - Once number of associated interpreters on both ends drops to 0, - the channel is actually marked as closed. The Python runtime - will garbage collect all closed channels. Note that "close()" is - automatically called when it is no longer used in the current - interpreter. + Once the number of associated interpreters on both ends drops + to 0, the channel is actually marked as closed. The Python + runtime will garbage collect all closed channels, though it may + not be immediately. Note that "close()" is automatically called + in behalf of the current interpreter when the channel is no longer + used in that interpreter. - This operation is idempotent. Return True if the current - interpreter was still associated with the receiving end of the - channel and False otherwise. + This operation is idempotent. Return True if "close()" has not + been called before by the current interpreter. ``SendChannel(id)``:: @@ -217,36 +224,26 @@ channels have no buffer. send(obj): - Send the object to the receiving end of the channel. Wait until - the object is received. If the channel does not support the - object then TypeError is raised. Currently only bytes are - supported. If the channel has been closed then EOFError is - raised. + Send the object to the receiving end of the channel. Wait until + the object is received. If the channel does not support the + object then TypeError is raised. Currently only bytes are + supported. + + If the channel is already closed (see the close() method) + then raise EOFError. If the channel isn't closed, but the current + interpreter already called the "close()" method (which drops its + association with the channel) then raise ValueError. send_nowait(obj): Send the object to the receiving end of the channel. If the - object is received then return True. Otherwise return False. - If the channel does not support the object then TypeError is - raised. If the channel has been closed then EOFError is raised. + object is received then return True. If not then return False. + Otherwise, this is the same as the "send()" method. close(): - No longer associate the current interpreter with the channel (on - the sending end). This is a noop if the interpreter isn't already - associated. Once an interpreter is no longer associated with the - channel, subsequent (or current) send() and recv() calls from that - interpreter will raise EOFError. - - Once number of associated interpreters on both ends drops to 0, - the channel is actually marked as closed. The Python runtime - will garbage collect all closed channels. Note that "close()" is - automatically called when it is no longer used in the current - interpreter. - - This operation is idempotent. Return True if the current - interpreter was still associated with the sending end of the - channel and False otherwise. + This is the same as "RecvChannel.close(), but applied to the + sending end of the channel. Examples From 1aff1f3706c423ff50b610d83c435933349a319b Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 19:29:32 +0000 Subject: [PATCH 05/26] Add an extra solution to the problem of resetting __main__. --- pep-0554.rst | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/pep-0554.rst b/pep-0554.rst index fcca7bd9fd9..137781344a7 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -731,13 +731,19 @@ desireable and you want to execute in a fresh ``__main__``. Also, you don't necessarily want to leak objects there that you aren't using any more. -Solutions include: +Note that the following work work right because it will clear too much +(e.g. ``__name__`` and the other "__dunder__" attributes:: + + interp.run('globals().clear()') + +Possible solutions include: * a ``create()`` arg to indicate resetting ``__main__`` after each ``run`` call * an ``Interpreter.reset_main`` flag to support opting in or out after the fact * an ``Interpreter.reset_main()`` method to opt in when desired +* ``importlib.util.reset_globals()`` [reset_globals]_ This isn't a critical feature initially. It can wait until later if desirable. @@ -848,6 +854,9 @@ References .. [bug-rate] https://mail.python.org/pipermail/python-ideas/2017-September/047094.html +.. [reset_globals] + https://mail.python.org/pipermail/python-dev/2017-September/149545.html + Copyright ========= From 1dd0050c7a3dbae4e11c21c5f0cdd049ac46bf98 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 19:42:27 +0000 Subject: [PATCH 06/26] Add a note about async. --- pep-0554.rst | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 137781344a7..93cfd12debf 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -765,6 +765,24 @@ would be a good candidate for the first effort at expanding the types that channels support. They aren't strictly necessary for the initial API. +Integration with async +---------------------- + +Per Antoine Pitrou [async]_:: + + Has any thought been given to how FIFOs could integrate with async + code driven by an event loop (e.g. asyncio)? I think the model of + executing several asyncio (or Tornado) applications each in their + own subinterpreter may prove quite interesting to reconcile multi- + core concurrency with ease of programming. That would require the + FIFOs to be able to synchronize on something an event loop can wait + on (probably a file descriptor?). + +A possible solution is to provide async implementations of the blocking +channel methods (``__next__()``, ``recv()``, and ``send()``). However, +the basic functionality of subinterpreters does not depend on async and +can be added later. + Rejected Ideas ============== @@ -857,6 +875,10 @@ References .. [reset_globals] https://mail.python.org/pipermail/python-dev/2017-September/149545.html +.. [async] + https://mail.python.org/pipermail/python-dev/2017-September/149420.html + https://mail.python.org/pipermail/python-dev/2017-September/149585.html + Copyright ========= From 66d5768fdcd359653c18a6596574f8325803c31d Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 20:12:03 +0000 Subject: [PATCH 07/26] Add an open question about the main thread. --- pep-0554.rst | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 93cfd12debf..b8c1aa18bf4 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -596,6 +596,31 @@ Switching to such an approach would help resolve questions of how passing bytes through channels will work once we isolate memory management in interpreters. +Disallow subinterpreters in the main thread? +-------------------------------------------- + +Currently in CPython, "we need a main \*thread\* in order to sensibly +manage the way signal handling works across different platforms". +[main-thread]_ The "main" interpreter is the initial one and the "main" +thread is the OS thread in which the main interpreter is running. + +Since signal handlers are part of the interpreter state, running a +subinterpreter in the main thread means that the main interpreter +can no longer properly handle signals (since it's effectively paused). + +Furthermore, running a subinterpreter in the main thread would +conceivably allow setting signal handlers on that interpreter, which +would likewise impact signal handling when that interpreter isn't +running or is running in a different thread. + +Ultimately, running subinterpreters in the main OS thread introduces +complications to the signal handling implementation. So it may make +the most sense to disallow running subinterpreters in the main thread. +Support for it could be considered later. The downside is that folks +wanting to try out subinterpreters would be required to take the extra +step of using threads. This could slow adoption and experimentation, +whereas without the restriction there's less of an obstacle. + Deferred Functionality ====================== @@ -872,6 +897,9 @@ References .. [bug-rate] https://mail.python.org/pipermail/python-ideas/2017-September/047094.html +.. [main-thread] + https://mail.python.org/pipermail/python-ideas/2017-September/047144.html + .. [reset_globals] https://mail.python.org/pipermail/python-dev/2017-September/149545.html From 42d44d958e2da4d2f177f90cc5dad30d0edb051a Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 20:17:07 +0000 Subject: [PATCH 08/26] Clarify about "execution namespace". --- pep-0554.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index b8c1aa18bf4..aea757646ee 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -79,9 +79,10 @@ The module also provides the following class: Run the provided Python source code in the interpreter. Any keyword arguments are added to the interpreter's execution - namespace. If any of the values are not supported for sharing - between interpreters then RuntimeError gets raised. Currently - only channels (see "create_channel()" below) are supported. + namespace (the interpreter's "__main__" module). If any of the + values are not supported for sharing between interpreters then + RuntimeError gets raised. Currently only channels (see + "create_channel()" below) are supported. This may not be called on an already running interpreter. Doing so results in a RuntimeError. From 4312d6436499308b6a3d9d0568d02544dcfb768f Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 20:18:39 +0000 Subject: [PATCH 09/26] Raise ValueError for unsupported values. --- pep-0554.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index aea757646ee..d675aa2aa7d 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -81,7 +81,7 @@ The module also provides the following class: keyword arguments are added to the interpreter's execution namespace (the interpreter's "__main__" module). If any of the values are not supported for sharing between interpreters then - RuntimeError gets raised. Currently only channels (see + ValueError gets raised. Currently only channels (see "create_channel()" below) are supported. This may not be called on an already running interpreter. Doing @@ -227,7 +227,7 @@ channels have no buffer. Send the object to the receiving end of the channel. Wait until the object is received. If the channel does not support the - object then TypeError is raised. Currently only bytes are + object then ValueError is raised. Currently only bytes are supported. If the channel is already closed (see the close() method) From 6363e64e0c142be8b82ab88209db46f57ed786c5 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 20:23:04 +0000 Subject: [PATCH 10/26] Clarify about "no longer used". --- pep-0554.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0554.rst b/pep-0554.rst index d675aa2aa7d..dfad2adca73 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -202,7 +202,7 @@ channels have no buffer. runtime will garbage collect all closed channels, though it may not be immediately. Note that "close()" is automatically called in behalf of the current interpreter when the channel is no longer - used in that interpreter. + used (i.e. has no references) in that interpreter. This operation is idempotent. Return True if "close()" has not been called before by the current interpreter. From 859fb018aad40a21691cb06e5c306d90a2ca98ae Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 20:30:45 +0000 Subject: [PATCH 11/26] Drop RecvChannel.__next__(). --- pep-0554.rst | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index dfad2adca73..1ee4ad44f69 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -165,11 +165,6 @@ channels have no buffer. The list of associated interpreters: those that have called the "recv()" or "__next__()" methods and haven't called "close()". - __next__(): - - Return the next object from the channel. If none have been sent - then wait until the next send. See the recv() method. - recv(): Return the next object from the channel. If none have been sent @@ -809,6 +804,15 @@ channel methods (``__next__()``, ``recv()``, and ``send()``). However, the basic functionality of subinterpreters does not depend on async and can be added later. +Support for iteration +--------------------- + +Supporting iteration on ``RecvChannel`` (via ``__iter__()`` or +``_next__()``) may be useful. A trivial implementation would use the +``recv()`` method, similar to how files do iteration. Since this isn't +a fundamental capability and has a simple analog, adding iteration +support can wait until later. + Rejected Ideas ============== From de9cc538d3abc7f6766aa15433a57856f9963cd4 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 20:34:55 +0000 Subject: [PATCH 12/26] Add a note about context managers. --- pep-0554.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 1ee4ad44f69..a27f19de429 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -813,6 +813,13 @@ Supporting iteration on ``RecvChannel`` (via ``__iter__()`` or a fundamental capability and has a simple analog, adding iteration support can wait until later. +Channel context managers +------------------------ + +Context manager support on ``RecvChannel`` and ``SendChannel`` may be +helpful. The implementation would be simple, wrapping a call to +``close()`` like files do. As with iteration, this can wait. + Rejected Ideas ============== From 7990474f7a790a05238bad0efecb863b00b4d479 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 20:38:22 +0000 Subject: [PATCH 13/26] Fix a typo. --- pep-0554.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0554.rst b/pep-0554.rst index a27f19de429..e070cfc41c6 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -584,7 +584,7 @@ Initial support for buffers in channels --------------------------------------- An alternative to support for bytes in channels in support for -read-only buffers (the PEP 3119 kind). Then ``recv()`` would return +read-only buffers (the PEP 3118 kind). Then ``recv()`` would return a memoryview to expose the buffer in a zero-copy way. This is similar to what ``multiprocessing.Connection`` supports. [mp-conn] From 8d9d4eb69cf51283c11a69c5bca1b03f0f5f61e9 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 20:38:42 +0000 Subject: [PATCH 14/26] Fix the note about timeouts. --- pep-0554.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index e070cfc41c6..c7beee43276 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -639,11 +639,13 @@ This suffers from the same problem as sharing objects between interpreters via queues. The minimal solution (running a source string) is sufficient for us to get the feature out where it can be explored. -timeout arg to pop() and push() -------------------------------- +timeout arg to recv() and send() +-------------------------------- Typically functions that have a ``block`` argument also have a -``timeout`` argument. We can add it later if needed. +``timeout`` argument. It sometimes makes sense to do likewise for +functions that otherwise block, like the channel ``recv()`` and +``send()`` methods. We can add it later if needed. get_main() ---------- From e46790abcd845b064238c97a1f5d1824d1e33f29 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 20:41:16 +0000 Subject: [PATCH 15/26] Fix a typo. --- pep-0554.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0554.rst b/pep-0554.rst index c7beee43276..9072ea0c995 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -754,7 +754,7 @@ desireable and you want to execute in a fresh ``__main__``. Also, you don't necessarily want to leak objects there that you aren't using any more. -Note that the following work work right because it will clear too much +Note that the following won't work right because it will clear too much (e.g. ``__name__`` and the other "__dunder__" attributes:: interp.run('globals().clear()') From efd0d9a194fdce6d4074a21ec7dfda92eb4714c0 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 20:59:27 +0000 Subject: [PATCH 16/26] Clarify about resetting __main__. --- pep-0554.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 9072ea0c995..3f9ebdf4a8a 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -768,6 +768,16 @@ Possible solutions include: * an ``Interpreter.reset_main()`` method to opt in when desired * ``importlib.util.reset_globals()`` [reset_globals]_ +Also note that reseting ``__main__`` does nothing about state stored +in other modules. So any solution would have to be clear about the +scope of what is being reset. Conceivably we could invent a mechanism +by which any (or every) module could be reset, unlike ``reload()`` +which does not clear the module before loading into it. Regardless, +since ``__main__`` is the execution namespace of the interpreter, +resetting it has a much more direct correlation to interpreters and +their dynamic state than does resetting other modules. So a more +generic module reset mechanism may prove unnecessary. + This isn't a critical feature initially. It can wait until later if desirable. From ac2f83261ee21ee7e0d4c697e6d13f14e48ec7fb Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 21:14:29 +0000 Subject: [PATCH 17/26] Add a note about pipes and queues. --- pep-0554.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 3f9ebdf4a8a..d1cc370cf73 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -832,6 +832,21 @@ Context manager support on ``RecvChannel`` and ``SendChannel`` may be helpful. The implementation would be simple, wrapping a call to ``close()`` like files do. As with iteration, this can wait. +Pipes and Queues +---------------- + +With the proposed object passing machanism of "channels", other similar +basic types aren't required to achieve the minimal useful functionality +of subinterpreters. Such types include pipes (like channels, but +one-to-one) and queues (like channels, but buffered). See below in +`Rejected Ideas` for more information. + +Even though these types aren't part of this proposal, they may still +be useful in the context of concurrency. Adding them later is entirely +reasonable. The could be trivially implemented as wrappers around +channels. Alternatively they could be implemented for efficiency at the +same low level as channels. + Rejected Ideas ============== From 5d73daa3964139800039eac553fc366a198339e2 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 21:30:06 +0000 Subject: [PATCH 18/26] Quote Nick about isolation. --- pep-0554.rst | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index d1cc370cf73..e064f2dba24 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -379,6 +379,27 @@ isolation within the same process. This can be leveraged in number of ways. Furthermore, subinterpreters provide a well-defined framework in which such isolation may extended. +Nick Coghlan explained some of the benefits through a comparison with +multi-processing [benefits]_:: + + [I] expect that communicating between subinterpreters is going + to end up looking an awful lot like communicating between + subprocesses via shared memory. + + The trade-off between the two models will then be that one still + just looks like a single process from the point of view of the + outside world, and hence doesn't place any extra demands on the + underlying OS beyond those required to run CPython with a single + interpreter, while the other gives much stricter isolation + (including isolating C globals in extension modules), but also + demands much more from the OS when it comes to its IPC + capabilities. + + The security risk profiles of the two approaches will also be quite + different, since using subinterpreters won't require deliberately + poking holes in the process isolation that operating systems give + you by default. + CPython has supported subinterpreters, with increasing levels of support, since version 1.5. While the feature has the potential to be a powerful tool, subinterpreters have suffered from neglect @@ -936,6 +957,9 @@ References .. [bug-rate] https://mail.python.org/pipermail/python-ideas/2017-September/047094.html +.. [benefits] + https://mail.python.org/pipermail/python-ideas/2017-September/047122.html + .. [main-thread] https://mail.python.org/pipermail/python-ideas/2017-September/047144.html From 2d478828737b29e45522374b049921e0d5c330de Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 21:44:57 +0000 Subject: [PATCH 19/26] Note possible solutions to exception leakage. --- pep-0554.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index e064f2dba24..1a41b36e83f 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -601,6 +601,15 @@ interpreters get better isolation relative to memory management (which is necessary to stop sharing the GIL between interpreters). So the semantics of how the exceptions propagate needs to be resolved. +Possible solutions: + +* convert at the boundary (e.g. ``subprocess.CalledProcessError``) +* wrap in a proxy at the boundary (including with support for + something like ``err.raise()`` to propagate the traceback). +* return the exception (or its proxy) from ``run()`` instead of + raising it +* return a result object (like ``subprocess`` does) + Initial support for buffers in channels --------------------------------------- From 7c18b9703a57de2d86d4dfe10c4603f4bd785eef Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 21:48:38 +0000 Subject: [PATCH 20/26] Use textwrap.dedent() in examples. --- pep-0554.rst | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index 1a41b36e83f..a7d819b3543 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -274,15 +274,15 @@ Pre-populate an interpreter :: interp = interpreters.create() - interp.run("""if True: + interp.run(tw.dedent(""" import some_lib import an_expensive_module some_lib.set_up() - """) + """)) wait_for_request() - interp.run("""if True: + interp.run(tw.dedent(""" some_lib.handle_request() - """) + """)) Handling an exception --------------------- @@ -291,9 +291,9 @@ Handling an exception interp = interpreters.create() try: - interp.run("""if True: + interp.run(tw.dedent(""" raise KeyError - """) + """)) except KeyError: print("got the error from the subinterpreter") @@ -305,12 +305,12 @@ Synchronize using a channel interp = interpreters.create() r, s = interpreters.create_channel() def run(): - interp.run("""if True: + interp.run(tw.dedent(""" reader.recv() print("during") reader.close() - """, - reader=r) + """), + reader=r)) t = threading.Thread(target=run) print('before') t.start() @@ -327,13 +327,13 @@ Sharing a file descriptor r1, s1 = interpreters.create_channel() r2, s2 = interpreters.create_channel() def run(): - interp.run("""if True: + interp.run(tw.dedent(""" fd = int.from_bytes( reader.recv(), 'big') for line in os.fdopen(fd): print(line) writer.send(b'') - """, + """), reader=r1, writer=s2) t = threading.Thread(target=run) t.start() @@ -349,19 +349,19 @@ Passing objects via pickle interp = interpreters.create() r, s = interpreters.create_channel() - interp.run("""if True: + interp.run(tw.dedent(""" import pickle - """, + """), reader=r) def run(): - interp.run("""if True: + interp.run(tw.dedent(""" data = reader.recv() while data: obj = pickle.loads(data) do_something(obj) data = reader.recv() reader.close() - """, + """), reader=r) t = threading.Thread(target=run) t.start() From a52e367a74b7e48e10fa84c068a888bb056e9c59 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 22:06:11 +0000 Subject: [PATCH 21/26] Add a note about distinguishing errors out of run(). --- pep-0554.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index a7d819b3543..2e0785bfdb7 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -877,6 +877,21 @@ reasonable. The could be trivially implemented as wrappers around channels. Alternatively they could be implemented for efficiency at the same low level as channels. +interpreters.RunFailedError +--------------------------- + +As currently proposed, ``Interpreter.run()`` offers you no way to +distinguish an error coming from sub-interpreter from any other +error in the current interpreter. Your only option would be to +explicitly wrap your ``run()`` call in a ``try: ... except Exception:``. + +If this is a problem in practice then would could add something like +``interpreters.RunFailedError`` and raise that in ``run()``, chaining +the actual error. + +Of course, this depends on how we resolve `Leaking exceptions across +interpreters`_. + Rejected Ideas ============== From f4fe10f05ed4ce21d9583b85e0f230c3ff404605 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 22:25:14 +0000 Subject: [PATCH 22/26] Add more options for not leaking exceptions. --- pep-0554.rst | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/pep-0554.rst b/pep-0554.rst index 2e0785bfdb7..d4915be2b9c 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -608,7 +608,11 @@ Possible solutions: something like ``err.raise()`` to propagate the traceback). * return the exception (or its proxy) from ``run()`` instead of raising it -* return a result object (like ``subprocess`` does) +* return a result object (like ``subprocess`` does) [result-object]_ +* throw the exception away and expect users to deal with unhandled + exceptions explicitly in the script they pass to ``run()`` + (they can pass error info out via channels); with threads you have + to do something similar Initial support for buffers in channels --------------------------------------- @@ -994,6 +998,9 @@ References https://mail.python.org/pipermail/python-dev/2017-September/149420.html https://mail.python.org/pipermail/python-dev/2017-September/149585.html +.. [result-object] + https://mail.python.org/pipermail/python-dev/2017-September/149562.html + Copyright ========= From 3fd9f9fb809f3e6219f45f0fda1d1fc88620311d Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 22:25:40 +0000 Subject: [PATCH 23/26] Add a note about explicitly passing channels to run(). --- pep-0554.rst | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index d4915be2b9c..7638635b63e 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -651,6 +651,34 @@ wanting to try out subinterpreters would be required to take the extra step of using threads. This could slow adoption and experimentation, whereas without the restriction there's less of an obstacle. +Pass channels explicitly to run()? +---------------------------------- + +Nick Coghlan suggested [explicit-channels]_ that we may want something more explicit than +the keyword args of ``run()`` (``**shared``):: + + The subprocess.run() comparison does make me wonder whether this + might be a more future-proof signature for Interpreter.run() though: + + def run(source_str, /, *, channels=None): + ... + + That way channels can be a namespace *specifically* for passing in + channels, and can be reported as such on RunResult. If we decide + to allow arbitrary shared objects in the future, or add flag options + like "reraise=True" to reraise exceptions from the subinterpreter + in the current interpreter, we'd have that ability, rather than + having the entire potential keyword namespace taken up for passing + shared objects. + +and:: + + It does occur to me that if we wanted to align with the way the + `runpy` module spells that concept, we'd call the option + `init_globals`, but I'm thinking it will be better to only allow + channels to be passed through directly, and require that everything + else be sent through a channel. + Deferred Functionality ====================== @@ -991,6 +1019,10 @@ References .. [main-thread] https://mail.python.org/pipermail/python-ideas/2017-September/047144.html +.. [explicit-channels] + https://mail.python.org/pipermail/python-dev/2017-September/149562.html + https://mail.python.org/pipermail/python-dev/2017-September/149565.html + .. [reset_globals] https://mail.python.org/pipermail/python-dev/2017-September/149545.html From d3c1cf7eb54c48443713804a4de8fc8bf6930b64 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 22:40:50 +0000 Subject: [PATCH 24/26] Add more notes about the "main" thread. --- pep-0554.rst | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index 7638635b63e..09b22dba60f 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -626,13 +626,22 @@ Switching to such an approach would help resolve questions of how passing bytes through channels will work once we isolate memory management in interpreters. +Does every interpreter think that their thread is the "main" thread? +-------------------------------------------------------------------- + +CPython's interpreter implementation identifies the OS thread in which +it was started as the "main" thread. The interpreter the has slightly +different behavior depending on if the current thread is the main one +or not. This presents a problem in cases where "main thread" is meant +to imply "main thread in the main interpreter" [main-thread]_, where +the main interpreter is the initial one. + Disallow subinterpreters in the main thread? -------------------------------------------- -Currently in CPython, "we need a main \*thread\* in order to sensibly -manage the way signal handling works across different platforms". -[main-thread]_ The "main" interpreter is the initial one and the "main" -thread is the OS thread in which the main interpreter is running. +This is a specific case of the above issue. Currently in CPython, +"we need a main \*thread\* in order to sensibly manage the way signal +handling works across different platforms". [main-thread]_ Since signal handlers are part of the interpreter state, running a subinterpreter in the main thread means that the main interpreter @@ -1018,6 +1027,7 @@ References .. [main-thread] https://mail.python.org/pipermail/python-ideas/2017-September/047144.html + https://mail.python.org/pipermail/python-dev/2017-September/149566.html .. [explicit-channels] https://mail.python.org/pipermail/python-dev/2017-September/149562.html From 150ac9eeaf1a1f7b1c7f14b4d816c6b4eb506781 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 23:02:03 +0000 Subject: [PATCH 25/26] Fill in a little info about other implementations. --- pep-0554.rst | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 09b22dba60f..cacbb3d5161 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -582,8 +582,18 @@ remove it. Alternate Python Implementations ================================ +I'll be soliciting feedback from the different Python implementors about +subinterpreter support. + +Multiple-interpter support in the major Python implementations: + TBD +* jython: yes [jython]_ +* ironpython: yes? +* pypy: maybe not? [pypy]_ +* micropython: ??? + Open Questions ============== @@ -1043,6 +1053,12 @@ References .. [result-object] https://mail.python.org/pipermail/python-dev/2017-September/149562.html +.. [jython] + https://mail.python.org/pipermail/python-ideas/2017-May/045771.html + +.. [pypy] + https://mail.python.org/pipermail/python-ideas/2017-September/046973.html + Copyright ========= From 51979396acc75ae0cb92d0b95e2eb20debde8459 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 22 Sep 2017 23:26:45 +0000 Subject: [PATCH 26/26] Do not mention being inspired by the threading module. --- pep-0554.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index cacbb3d5161..34ba6a3f4c1 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -29,9 +29,8 @@ Proposal The ``interpreters`` module will be added to the stdlib. It will provide a high-level interface to subinterpreters and wrap the low-level -``_interpreters`` module. The proposed API is inspired by the -``threading`` module. See the `Examples`_ section for concrete usage -and use cases. +``_interpreters`` module. See the `Examples`_ section for concrete +usage and use cases. API for interpreters --------------------