From 532a67d28e351eb4c99e2750e3132bed61512b55 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 14:06:51 +0000 Subject: [PATCH 01/18] Re-structure the PEP. --- pep-0554.rst | 225 ++++++++++++++++++++++++++++----------------------- 1 file changed, 126 insertions(+), 99 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index 0d2d8795c45..e23c74bc25e 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -12,86 +12,15 @@ Post-History: Abstract ======== -This proposal introduces the stdlib ``interpreters`` module. It exposes -the basic functionality of subinterpreters that already exists in the -C-API. Each subinterpreter runs with its own state (see -``Interpreter Isolation`` below). The module will be "provisional", as -described by PEP 411. - - -Rationale -========= - -Running code in multiple interpreters provides a useful level of -isolation within the same process. This can be leveraged in number -of ways. Furthermore, subinterpreters provide a well-defined framework -in which such isolation may extended. - CPython has supported subinterpreters, with increasing levels of -support, since version 1.5. While the feature has the potential -to be a powerful tool, subinterpreters have suffered from neglect -because they are not available directly from Python. Exposing the -existing functionality in the stdlib will help reverse the situation. - -This proposal is focused on enabling the fundamental capability of -multiple isolated interpreters in the same Python process. This is a -new area for Python so there is relative uncertainly about the best -tools to provide as companions to subinterpreters. Thus we minimize -the functionality we add in the proposal as much as possible. - -Concerns --------- - -* "subinterpreters are not worth the trouble" - -Some have argued that subinterpreters do not add sufficient benefit -to justify making them an official part of Python. Adding features -to the language (or stdlib) has a cost in increasing the size of -the language. So it must pay for itself. In this case, subinterpreters -provide a novel concurrency model focused on isolated threads of -execution. Furthermore, they present an opportunity for changes in -CPython that will allow simulateous use of multiple CPU cores (currently -prevented by the GIL). - -Alternatives to subinterpreters include threading, async, and -multiprocessing. Threading is limited by the GIL and async isn't -the right solution for every problem (nor for every person). -Multiprocessing is likewise valuable in some but not all situations. -Direct IPC (rather than via the multiprocessing module) provides -similar benefits but with the same caveat. - -Notably, subinterpreters are not intended as a replacement for any of -the above. Certainly they overlap in some areas, but the benefits of -subinterpreters include isolation and (potentially) performance. In -particular, subinterpreters provide a direct route to an alternate -concurrency model (e.g. CSP) which has found success elsewhere and -will appeal to some Python users. That is the core value that the -``interpreters`` module will provide. - -* "stdlib support for subinterpreters adds extra burden - on C extension authors" +support, since version 1.5. The feature has been available via the +C-API. [c-api]_ Subinterpreters operate in relative isolation from one +another (see ``Interpreter Isolation`` below), which provides the basis +for an alternative concurrency model. -In the ``Interpreter Isolation`` section below we identify ways in -which isolation in CPython's subinterpreters is incomplete. Most -notable is extension modules that use C globals to store internal -state. PEP 3121 and PEP 489 provide a solution for most of the -problem, but one still remains. [petr-c-ext]_ Until that is resolved, -C extension authors will face extra difficulty to support -subinterpreters. - -Consequently, projects that publish extension modules may face an -increased maintenance burden as their users start using subinterpreters, -where their modules may break. This situation is limited to modules -that use C globals (or use libraries that use C globals) to store -internal state. - -Ultimately this comes down to a question of how often it will be a -problem in practice: how many projects would be affected, how often -their users will be affected, what the additional maintenance burden -will be for projects, and what the overall benefit of subinterpreters -is to offset those costs. The position of this PEP is that the actual -extra maintenance burden will be small and well below the threshold at -which subinterpreters are worth it. +This proposal introduces the stdlib ``interpreters`` module. The module +will be provisional (see ``Provisional Status`` below). It exposes the +basic functionality of subinterpreters already provided by the C-API. Proposal @@ -265,41 +194,97 @@ minimal and initially restricting the supported types helps us avoid further exposing any underlying complexity to Python users. -Deferred Functionality -====================== +Examples +======== -In the interest of keeping this proposal minimal, the following -functionality has been left out for future consideration. Note that -this is not a judgement against any of said capability, but rather a -deferment. That said, each is arguably valid. +TBD -Interpreter.call() ------------------- -It would be convenient to run existing functions in subinterpreters -directly. ``Interpreter.run()`` could be adjusted to support this or -a ``call()`` method could be added:: +Rationale +========= - Interpreter.call(f, *args, **kwargs) +Running code in multiple interpreters provides a useful level of +isolation within the same process. This can be leveraged in number +of ways. Furthermore, subinterpreters provide a well-defined framework +in which such isolation may extended. -This suffers from the same problem as sharing objects between -interpreters via queues. The minimal solution (running a source string) -is sufficient for us to get the feature out where it can be explored. +CPython has supported subinterpreters, with increasing levels of +support, since version 1.5. While the feature has the potential +to be a powerful tool, subinterpreters have suffered from neglect +because they are not available directly from Python. Exposing the +existing functionality in the stdlib will help reverse the situation. -timeout arg to pop() and push() -------------------------------- +This proposal is focused on enabling the fundamental capability of +multiple isolated interpreters in the same Python process. This is a +new area for Python so there is relative uncertainly about the best +tools to provide as companions to subinterpreters. Thus we minimize +the functionality we add in the proposal as much as possible. -Typically functions that have a ``block`` argument also have a -``timeout`` argument. We can add it later if needed. +Concerns +-------- +* "subinterpreters are not worth the trouble" -Interpreter Isolation +Some have argued that subinterpreters do not add sufficient benefit +to justify making them an official part of Python. Adding features +to the language (or stdlib) has a cost in increasing the size of +the language. So it must pay for itself. In this case, subinterpreters +provide a novel concurrency model focused on isolated threads of +execution. Furthermore, they present an opportunity for changes in +CPython that will allow simulateous use of multiple CPU cores (currently +prevented by the GIL). + +Alternatives to subinterpreters include threading, async, and +multiprocessing. Threading is limited by the GIL and async isn't +the right solution for every problem (nor for every person). +Multiprocessing is likewise valuable in some but not all situations. +Direct IPC (rather than via the multiprocessing module) provides +similar benefits but with the same caveat. + +Notably, subinterpreters are not intended as a replacement for any of +the above. Certainly they overlap in some areas, but the benefits of +subinterpreters include isolation and (potentially) performance. In +particular, subinterpreters provide a direct route to an alternate +concurrency model (e.g. CSP) which has found success elsewhere and +will appeal to some Python users. That is the core value that the +``interpreters`` module will provide. + +* "stdlib support for subinterpreters adds extra burden + on C extension authors" + +In the ``Interpreter Isolation`` section below we identify ways in +which isolation in CPython's subinterpreters is incomplete. Most +notable is extension modules that use C globals to store internal +state. PEP 3121 and PEP 489 provide a solution for most of the +problem, but one still remains. [petr-c-ext]_ Until that is resolved, +C extension authors will face extra difficulty to support +subinterpreters. + +Consequently, projects that publish extension modules may face an +increased maintenance burden as their users start using subinterpreters, +where their modules may break. This situation is limited to modules +that use C globals (or use libraries that use C globals) to store +internal state. + +Ultimately this comes down to a question of how often it will be a +problem in practice: how many projects would be affected, how often +their users will be affected, what the additional maintenance burden +will be for projects, and what the overall benefit of subinterpreters +is to offset those costs. The position of this PEP is that the actual +extra maintenance burden will be small and well below the threshold at +which subinterpreters are worth it. + + +About Subinterpreters ===================== +Interpreter Isolation +--------------------- + CPython's interpreters are intended to be strictly isolated from each other. Each interpreter has its own copy of all modules, classes, functions, and variables. The same applies to state in C, including in -extension modules. The CPython C-API docs explain more. [c-api]_ +extension modules. The CPython C-API docs explain more. [caveats]_ However, there are ways in which interpreters share some state. First of all, some process-global state remains shared, like file descriptors. @@ -314,6 +299,11 @@ Finally, some potential isolation is missing due to the current design of CPython. This includes the GIL and memory management. Improvements are currently going on to address gaps in this area. +Existing Usage +-------------- + +TBD + Provisional Status ================== @@ -326,10 +316,47 @@ before the 3.8 release whether to keep it provisional, graduate it, or remove it. +Alternate Python Implementations +================================ + +TBD + + +Deferred Functionality +====================== + +In the interest of keeping this proposal minimal, the following +functionality has been left out for future consideration. Note that +this is not a judgement against any of said capability, but rather a +deferment. That said, each is arguably valid. + +Interpreter.call() +------------------ + +It would be convenient to run existing functions in subinterpreters +directly. ``Interpreter.run()`` could be adjusted to support this or +a ``call()`` method could be added:: + + Interpreter.call(f, *args, **kwargs) + +This suffers from the same problem as sharing objects between +interpreters via queues. The minimal solution (running a source string) +is sufficient for us to get the feature out where it can be explored. + +timeout arg to pop() and push() +------------------------------- + +Typically functions that have a ``block`` argument also have a +``timeout`` argument. We can add it later if needed. + + References ========== .. [c-api] + https://docs.python.org/3/c-api/init.html#sub-interpreter-support + +.. [caveats] https://docs.python.org/3/c-api/init.html#bugs-and-caveats .. [petr-c-ext] From 1fa43acf4507dbdcd3563ee83d5c99391098ae6c Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 14:29:57 +0000 Subject: [PATCH 02/18] Fix internal links. --- pep-0554.rst | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index e23c74bc25e..8531f418842 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -14,13 +14,14 @@ Abstract CPython has supported subinterpreters, with increasing levels of support, since version 1.5. The feature has been available via the -C-API. [c-api]_ Subinterpreters operate in relative isolation from one -another (see ``Interpreter Isolation`` below), which provides the basis -for an alternative concurrency model. +C-API. [c-api]_ Subinterpreters operate in +`relative isolation from one another `_, which +provides the basis for an +`alternative concurrency model `_. This proposal introduces the stdlib ``interpreters`` module. The module -will be provisional (see ``Provisional Status`` below). It exposes the -basic functionality of subinterpreters already provided by the C-API. +will be `provisional `_. It exposes the basic +functionality of subinterpreters already provided by the C-API. Proposal @@ -252,7 +253,7 @@ will appeal to some Python users. That is the core value that the * "stdlib support for subinterpreters adds extra burden on C extension authors" -In the ``Interpreter Isolation`` section below we identify ways in +In the `Interpreter Isolation`_ section below we identify ways in which isolation in CPython's subinterpreters is incomplete. Most notable is extension modules that use C globals to store internal state. PEP 3121 and PEP 489 provide a solution for most of the @@ -278,6 +279,8 @@ which subinterpreters are worth it. About Subinterpreters ===================== +.. isolation: + Interpreter Isolation --------------------- @@ -299,12 +302,21 @@ Finally, some potential isolation is missing due to the current design of CPython. This includes the GIL and memory management. Improvements are currently going on to address gaps in this area. +.. concurrency: + +Concurrency +----------- + +TBD + Existing Usage -------------- TBD +.. provisional: + Provisional Status ================== From 06a73eefd83431b5e675d95d69eaa2e458cb4b5f Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 15:03:28 +0000 Subject: [PATCH 03/18] Add CSP link. --- pep-0554.rst | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index 8531f418842..7b1373f2964 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -171,8 +171,9 @@ About FIFOs Subinterpreters are inherently isolated (with caveats explained below), in contrast to threads. This enables a different concurrency model than -currently exists in Python. CSP (Communicating Sequential Processes), -upon which Go's concurrency is based, is one example of this model. +currently exists in Python. +`Communicating Sequential Processes`_ (CSP), upon which Go's concurrency +is based, is one example of this model. A key component of this approach to concurrency is message passing. So providing a message/object passing mechanism alongside ``Interpreter`` @@ -368,6 +369,12 @@ References .. [c-api] https://docs.python.org/3/c-api/init.html#sub-interpreter-support +.. _Communicating Sequential Processes: + +.. [CSP] + https://en.wikipedia.org/wiki/Communicating_sequential_processes + https://github.com/futurecore/python-csp + .. [caveats] https://docs.python.org/3/c-api/init.html#bugs-and-caveats From e3b168a22695d02df2b6d7a21f52e74cb47d9fb8 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 15:14:06 +0000 Subject: [PATCH 04/18] Drop get_main(). --- pep-0554.rst | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index 7b1373f2964..8b8add033b3 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -42,10 +42,6 @@ The module provides the following functions: Return the currently running interpreter. -``get_main()``:: - - Return the main interpreter. - ``create()``:: Initialize a new Python interpreter and return it. The @@ -362,6 +358,18 @@ timeout arg to pop() and push() Typically functions that have a ``block`` argument also have a ``timeout`` argument. We can add it later if needed. +get_main() +---------- + +CPython has a concept of a "main" interpreter. This is the initial +interpreter created during CPython's runtime initialization. It may +be useful to identify the main interpreter. For instance, the main +interpreter should not be destroyed. However, for the basic +functionality of a high-level API a ``get_main()`` function is not +necessary. Furthermore, there is no requirement that a Python +implementation have a concept of a main interpreter. So until there's +a clear need we'll leave ``get_main()`` out. + References ========== From 2181af3d667c204048afb5253a976587b650e3cb Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 16:41:50 +0000 Subject: [PATCH 05/18] Enumerate shared state. --- pep-0554.rst | 39 ++++++++++++++++++++++++++++++++++----- 1 file changed, 34 insertions(+), 5 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index 8b8add033b3..eaca20ab0ae 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -287,17 +287,33 @@ functions, and variables. The same applies to state in C, including in extension modules. The CPython C-API docs explain more. [caveats]_ However, there are ways in which interpreters share some state. First -of all, some process-global state remains shared, like file descriptors. +of all, some process-global state remains shared: + +* file descriptors +* builtin types (e.g. dict, bytes) +* singletons (e.g. None) +* underlying static module data (e.g. functions) for + builtin/extension/frozen modules + There are no plans to change this. Second, some isolation is faulty due to bugs or implementations that did not take subinterpreters into account. This includes things like -at-exit handlers and extension modules that rely on C globals. In these -cases bugs should be opened (some are already). +extension modules that rely on C globals. [cryptography]_ In these +cases bugs should be opened (some are already): + +* readline module hook functions (http://bugs.python.org/issue4202) +* memory leaks on re-init (http://bugs.python.org/issue21387) Finally, some potential isolation is missing due to the current design -of CPython. This includes the GIL and memory management. Improvements -are currently going on to address gaps in this area. +of CPython. Improvements are currently going on to address gaps in this +area: + +* interpreters share the GIL +* interpreters share memory management (e.g. allocators, gc) +* GC is not run per-interpreter [global-gc]_ +* at-exit handlers are not run per-interpreter [global-atexit]_ +* extensions using the ``PyGILState_*`` API are incompatible [gilstate]_ .. concurrency: @@ -390,6 +406,19 @@ References https://mail.python.org/pipermail/import-sig/2016-June/001062.html https://mail.python.org/pipermail/python-ideas/2016-April/039748.html +.. [cryptography] + https://github.com/pyca/cryptography/issues/2299 + +.. [global-gc] + http://bugs.python.org/issue24554 + +.. [gilstate] + https://bugs.python.org/issue10915 + http://bugs.python.org/issue15751 + +.. [global-atexit] + https://bugs.python.org/issue6531 + Copyright ========= From db1bd7783ac909a7a76bc6cbce8e9d0632b11150 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 17:37:17 +0000 Subject: [PATCH 06/18] Clarify about subinterpreters + threads. --- pep-0554.rst | 53 +++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 40 insertions(+), 13 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index eaca20ab0ae..817b5f160e7 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -30,7 +30,8 @@ Proposal The ``interpreters`` module will be added to the stdlib. It will provide a high-level interface to subinterpreters and wrap the low-level ``_interpreters`` module. The proposed API is inspired by the -``threading`` module. +``threading`` module. See the `Examples`_ section for concrete usage +and use cases. The module provides the following functions: @@ -46,7 +47,9 @@ The module provides the following functions: Initialize a new Python interpreter and return it. The interpreter will be created in the current thread and will remain - idle until something is run in it. + idle until something is run in it. The interpreter may be used + in any thread and will run in whichever thread calls + ``interp.run()``. The module also provides the following classes: @@ -59,23 +62,40 @@ The module also provides the following classes: is_running(): Return whether or not the interpreter is currently executing code. + Calling this on the current interpreter will always return True. destroy(): - Finalize and destroy the interpreter. If called on a running - interpreter then raise RuntimeError in the interpreter that - called ``destroy()``. + Finalize and destroy the interpreter. - run(code): + This may not be called on an already running interpreter. Doing + so results in a RuntimeError. - Run the provided Python code in the interpreter, in the current - OS thread. If the interpreter is already running then raise - RuntimeError in the interpreter that called ``run()``. + run(source_text): - The current interpreter (which called ``run()``) will block until - the subinterpreter finishes running the requested code. Any - uncaught exception in that code will bubble up to the current - interpreter. + Run the provided Python source code in the interpreter. + + This may not be called on an already running interpreter. Doing + so results in a RuntimeError. + + A "run()" call is quite similar to any other function call. Once + it completes, the code that called "run()" continues executing + (in the original interpreter). Likewise, if there is any uncaught + exception, it propagates into the code where "run()" was called. + + The big difference is that "run()" executes the code in an + entirely different interpreter, with entirely separate state. + The state of the current interpreter in the current OS thread + is swapped out with the state of the target interpreter (the one + that will execute the code). When the target finishes executing, + the original interpreter gets swapped back in and its execution + resumes. + + So calling "run()" will effectively cause the current Python + thread to pause. Sometimes you won't want that pause, in which + case you should make the "run()" call in another thread. To do + so, add a function that calls "run()" and then run that function + in a normal "threading.Thread". Supported code: source text. @@ -386,6 +406,13 @@ necessary. Furthermore, there is no requirement that a Python implementation have a concept of a main interpreter. So until there's a clear need we'll leave ``get_main()`` out. +Interpreter.run_in_thread() +--------------------------- + +This method would make a ``run()`` call for you in a thread. Doing this +using only ``threading.Thread`` and ``run()`` is relatively trivial so +we've left it out. + References ========== From 96640a49926cadfe1797e565e7f15c16ec09d8ac Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 17:39:03 +0000 Subject: [PATCH 07/18] Fix a typo. --- pep-0554.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0554.rst b/pep-0554.rst index 817b5f160e7..d7e812e8a44 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -71,7 +71,7 @@ The module also provides the following classes: This may not be called on an already running interpreter. Doing so results in a RuntimeError. - run(source_text): + run(source_str): Run the provided Python source code in the interpreter. From 76054683d23fa89509be28f692ae742816b1b5c4 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 17:46:34 +0000 Subject: [PATCH 08/18] Add a note about synchronization primitives. --- pep-0554.rst | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index d7e812e8a44..2fd8975f292 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -413,6 +413,18 @@ This method would make a ``run()`` call for you in a thread. Doing this using only ``threading.Thread`` and ``run()`` is relatively trivial so we've left it out. +Synchronization Primitives +-------------------------- + +The ``threading`` module provides a number of synchronization primitives +for coordinating concurrent operations. This is especially necessary +due to the shared-state nature of threading. In contrast, +subinterpreters do not share state. Data sharing is restricted to +FIFOs, which do away with the need for explicit synchronization. If +any sort of opt-in shared state support is added to subinterpreters in +the future, that same effort can introduce synchronization primitives +to meet that need. + References ========== From 88382715e894a0309980ebbb8693d41c7c67ffb6 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 17:55:40 +0000 Subject: [PATCH 09/18] Defer more ideas. --- pep-0554.rst | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 2fd8975f292..529ae586d0f 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -425,6 +425,23 @@ any sort of opt-in shared state support is added to subinterpreters in the future, that same effort can introduce synchronization primitives to meet that need. +CSP Library +----------- + +A ``csp`` module would not be a large step away from the functionality +provided by this PEP. However, adding such a module is outside the +minimalist goals of this proposal. + +Syntactic Support +----------------- + +The ``Go`` language provides a concurrency model based on CSP, so +it's similar to the concurrency model that subinterpreters support. +``Go`` provides syntactic support, as well several builtin concurrency +primitives, to make concurrency a first-class feature. Conceivably, +similar syntactic (and builtin) support could be added to Python using +subinterpreters. However, that is *way* outside the scope of this PEP! + References ========== From 574c9cceb022acb0ab19465e1323a1df8f7b1f86 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 18:05:10 +0000 Subject: [PATCH 10/18] Fill in the existing usage section. --- pep-0554.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/pep-0554.rst b/pep-0554.rst index 529ae586d0f..b3af8e67a3c 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -345,7 +345,12 @@ TBD Existing Usage -------------- -TBD +Subinterpreters are not a widely used feature. In fact, the only +documented case of wide-spread usage is +`mod_wsgi `_. On the one +hand, this case provides confidence that existing subinterpreter support +is relatively stable. On the other hand, there isn't much of a sample +size from which to judge the utility of the feature. .. provisional: From 6e2d1b1c7ed7591de0cfb2a07529febd8167889c Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 18:06:47 +0000 Subject: [PATCH 11/18] Drop unused anchors. --- pep-0554.rst | 6 ------ 1 file changed, 6 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index b3af8e67a3c..941a4d2c91a 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -296,8 +296,6 @@ which subinterpreters are worth it. About Subinterpreters ===================== -.. isolation: - Interpreter Isolation --------------------- @@ -335,8 +333,6 @@ area: * at-exit handlers are not run per-interpreter [global-atexit]_ * extensions using the ``PyGILState_*`` API are incompatible [gilstate]_ -.. concurrency: - Concurrency ----------- @@ -353,8 +349,6 @@ is relatively stable. On the other hand, there isn't much of a sample size from which to judge the utility of the feature. -.. provisional: - Provisional Status ================== From a8a1b8b1c708ef241323bca5356fe3090b078a86 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 18:24:55 +0000 Subject: [PATCH 12/18] Fill in the concurrency section. --- pep-0554.rst | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index 941a4d2c91a..16191e77382 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -186,10 +186,9 @@ About FIFOs ----------- Subinterpreters are inherently isolated (with caveats explained below), -in contrast to threads. This enables a different concurrency model than -currently exists in Python. -`Communicating Sequential Processes`_ (CSP), upon which Go's concurrency -is based, is one example of this model. +in contrast to threads. This enables `a different concurrency model +`_ than currently exists in Python. +`Communicating Sequential Processes`_ (CSP) is the prime example. A key component of this approach to concurrency is message passing. So providing a message/object passing mechanism alongside ``Interpreter`` @@ -336,7 +335,16 @@ area: Concurrency ----------- -TBD +Concurrency is a challenging area of software development. Decades of +research and practice have led to a wide variety of concurrency models, +each with different goals. Most center on correctness and usability. + +One class of concurrency models focuses on isolated threads of +execution that interoperate through some message passing scheme. A +notable example is `Communicating Sequential Processes`_ (CSP), upon +which Go's concurrency is based. The isolation inherent to +subinterpreters makes them well-suited to this approach. + Existing Usage -------------- From 668609c38bd4d872da85e8a5342eeab7cde725cd Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 18:33:11 +0000 Subject: [PATCH 13/18] Add a note about interp state between runs. --- pep-0554.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 16191e77382..12e43f248b5 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -97,6 +97,12 @@ The module also provides the following classes: so, add a function that calls "run()" and then run that function in a normal "threading.Thread". + Note that interpreter's state is never reset, neither before + "run()" executes the code nor after. Thus the interpreter + state is preserved between calls to "run()". This includes + "sys.modules", the "builtins" module, and the internal state + of C extension modules. + Supported code: source text. get_fifo(name): From 6bf53bbc3f4416b74fb820e5a903b751d331aee6 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 18:40:37 +0000 Subject: [PATCH 14/18] Stub out some examples. --- pep-0554.rst | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 12e43f248b5..12762721b94 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -220,6 +220,44 @@ further exposing any underlying complexity to Python users. Examples ======== +Run isolated code +----------------- + +TBD + +Run in a thread +--------------- + +TBD + +Pre-populate an interpreter +--------------------------- + +TBD + +Handling an exception +--------------------- + +TBD + +Interpreter pools +----------------- + +TBD + +Using a FIFO +------------ + +TBD + +Sharing a file descriptor +------------------------- + +TBD + +Passing objects via pickle +-------------------------- + TBD From 154052d8b5f195c66b40cf4ee670ef3beae5a592 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 18:43:25 +0000 Subject: [PATCH 15/18] Add a note about multi-processing. --- pep-0554.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 12762721b94..4b0edd1143d 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -493,6 +493,14 @@ primitives, to make concurrency a first-class feature. Conceivably, similar syntactic (and builtin) support could be added to Python using subinterpreters. However, that is *way* outside the scope of this PEP! +Multiprocessing +--------------- + +The ``multiprocessing`` module could support subinterpreters in the same +way it supports threads and processes. In fact, the module's +maintainer, Davin Potts, has indicated this is a reasonable feature +request. However, it is outside the narrow scope of this PEP. + References ========== From c3cbcf834ad3caee78e897873070b297e762a347 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 19:08:32 +0000 Subject: [PATCH 16/18] Fill in some of the examples. --- pep-0554.rst | 71 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 57 insertions(+), 14 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index 4b0edd1143d..024f8a7972d 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -223,32 +223,75 @@ Examples Run isolated code ----------------- -TBD +:: + + interp = interpreters.create() + print('before') + interp.run('print("during")') + print('after') Run in a thread --------------- -TBD +:: + + interp = interpreters.create() + def run(): + interp.run('print("during")') + t = threading.Thread(target=run) + print('before') + t.start() + print('after') Pre-populate an interpreter --------------------------- -TBD +:: + + interp = interpreters.create() + interp.run("""if True: + import some_lib + import an_expensive_module + some_lib.set_up() + """) + wait_for_request() + interp.run("""if True: + some_lib.handle_request() + """) Handling an exception --------------------- -TBD - -Interpreter pools ------------------ - -TBD - -Using a FIFO ------------- - -TBD +:: + + interp = interpreters.create() + try: + interp.run("""if True: + raise KeyError + """) + except KeyError: + print("got the error from the subinterpreter") + +Synchronize using a FIFO +------------------------ + +:: + + interp = interpreters.create() + writer = interp.add_recv_fifo('spam') + def run(): + interp.run("""if True: + import interpreters + interp = interpreters.get_current() + reader = interp.get_fifo('spam') + reader.pop() + print("during") + """) + t = threading.Thread(target=run) + print('before') + t.start() + print('after') + writer.push(None) Sharing a file descriptor ------------------------- From 6bf6e06bd0b569df1e9b395eb156074327a0f3fd Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 19:14:17 +0000 Subject: [PATCH 17/18] Add a note about __main__. --- pep-0554.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/pep-0554.rst b/pep-0554.rst index 024f8a7972d..34e8ea3d6a9 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -103,6 +103,13 @@ The module also provides the following classes: "sys.modules", the "builtins" module, and the internal state of C extension modules. + Also note that "run()" executes in the namespace of the "__main__" + module, just like scripts, the REPL, "-m", and "-c". Just as + the interpreter's state is not ever reset, the "__main__" module + is never reset. You can imagine concatenating the code from each + "run()" call into one long script. This is the same as how the + REPL operates. + Supported code: source text. get_fifo(name): From 2c95da229f99507c84e04940f343ca8c7f1f6fab Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 12 Sep 2017 19:29:12 +0000 Subject: [PATCH 18/18] Fill out the remaining examples. --- pep-0554.rst | 47 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 45 insertions(+), 2 deletions(-) diff --git a/pep-0554.rst b/pep-0554.rst index 34e8ea3d6a9..b8b47cdc0c3 100644 --- a/pep-0554.rst +++ b/pep-0554.rst @@ -303,12 +303,55 @@ Synchronize using a FIFO Sharing a file descriptor ------------------------- -TBD +:: + + interp = interpreters.create() + writer = interp.add_recv_fifo('spam') + reader = interp.add_send_fifo('done') + def run(): + interp.run("""if True: + import interpreters + interp = interpreters.get_current() + reader = interp.get_fifo('spam') + writer = interp.get_fifo('done') + fd = reader.pop() + for line in os.fdopen(fd): + print(line) + writer.push(None) + """) + t = threading.Thread(target=run) + t.start() + with open('spamspamspam') as infile: + writer.push(infile.fileno()) + reader.pop() Passing objects via pickle -------------------------- -TBD +:: + + interp = interpreters.create() + writer = interp.add_recv_fifo('spam') + interp.run("""if True: + import pickle + import interpreters + interp = interpreters.get_current() + reader = interp.get_fifo('spam') + """) + def run(): + interp.run("""if True: + data = reader.pop() + while data is not None: + obj = pickle.loads(data) + do_something(obj) + data = reader.pop() + """) + t = threading.Thread(target=run) + t.start() + for obj in input: + data = pickle.dumps(obj) + writer.push(data) + writer.push(None) Rationale