Skip to content

asyncio with two interpreter instances #91375

Closed
Closed
asyncio with two interpreter instances#91375
Assignees
erlend-aaslandkumaraditya303
Labels
3.12only security fixestopic-asynciotopic-subinterpreterstype-bugAn unexpected behavior, bug, or error

Description

@mbadaire
mbadaire
mannequin
opened on Apr 4, 2022
BPO 47219
Nosy @asvetlov, @ericsnowcurrently, @1st1

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-04-04.19:57:08.003>
labels = ['type-bug', 'expert-asyncio']
title = 'asyncio with two interpreter instances'
updated_at = <Date 2022-04-04.20:38:00.422>
user = 'https://bugs.python.org/mbadaire'

bugs.python.org fields:

activity = <Date 2022-04-04.20:38:00.422>
actor = 'JelleZijlstra'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['asyncio']
creation = <Date 2022-04-04.19:57:08.003>
creator = 'mbadaire'
dependencies = []
files = []
hgrepos = []
issue_num = 47219
keywords = []
message_count = 1.0
messages = ['416694']
nosy_count = 4.0
nosy_names = ['asvetlov', 'eric.snow', 'yselivanov', 'mbadaire']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue47219'
versions = []

Activity

  1. mbadaire commented on Apr 4, 2022

    @mbadaire
    mbadairemannequin
    on Apr 4, 2022
    MannequinAuthor

    Hi,
    I have an issue when using asyncio and two interpreter instances each launched and used in a seperated thread.
    I am getting a asyncio loop for each thread .However asyncio is getting me the same loop because of this code in get_running_loop. Indeed when I have two interpreter, the ts_id would be the same for both my threads and therefore I will get the cached value of the first thread. cached_running_holder being static, it is the same value for all instances of interpreter.
    Maybe we should check if we are in the same interpreter or same thread ,.. I am not sure how it could be fixed.

    _asynciomodule.c:
get_running_loop(PyObject **loop)
{
    PyObject *rl;

    PyThreadState *ts = _PyThreadState_GET();
    uint64_t ts_id = PyThreadState_GetID(ts);
    if (ts_id == cached_running_holder_tsid && cached_running_holder != NULL) {

    If it does not make sense, I have some sample code but it is not just 10 lines.

  2. mbadaire
    added
    topic-asyncio
    type-bugAn unexpected behavior, bug, or error
    on Apr 4, 2022
  3. ezio-melotti
    transferred this issue fromon Apr 10, 2022

  4. pipiche38 commented on May 6, 2022

    @pipiche38
    pipiche38
    on May 6, 2022

    hello, is there anything we can do to get some attention ?
    This is a blocking issue in our project which spawn several instances from a Python embedded framework

    Thanks in advance

  5. ericsnowcurrently commented on May 6, 2022

    @ericsnowcurrently
    ericsnowcurrently
    on May 6, 2022
    Member

    It looks like asyncio has not been updated to support use in multiple interpreters -- the module uses a whole bunch of global variables to store state. Until that is dealt with, asyncio cannot be used reliably with multiple interpreters. See PEP 687. @erlend-aasland

    Also note that this is unlikely to be considered a bug, so no backports. Likewise it is unlikely to happen for 3.11 since the feature freeze is literally today. Thus the soonest you may see a fix is 3.12 (Fall 2023), unfortunately.

  6. erlend-aasland commented on May 7, 2022

    @erlend-aasland
    erlend-aasland
    on May 7, 2022
    Contributor

    Yeah, as soon as PEP 687 land, we can start the process of implementing this. I believe I've got a WIP branch lying around already.

  7. pipiche38 commented on May 7, 2022

    @pipiche38
    pipiche38
    on May 7, 2022

    Thanks for the update, we will be patient. just a shame not to have back port, but understood the amount of work.
    Now, we would be really please to test it when you want .

    Potential testing platforms OS: raspian or fedora

  8. erlend-aasland commented on May 7, 2022

    @erlend-aasland
    erlend-aasland
    on May 7, 2022
    Contributor

    I can push it to my fork next week, so you can play with it.

  9. kumaraditya303 commented on May 24, 2022

    @kumaraditya303
    kumaraditya303
    on May 24, 2022
    Contributor

    One workaround currently is to not use _asyncio speed up module but to use pure python version as that does not has this limitation.

  10. pipiche38 commented on May 24, 2022

    @pipiche38
    pipiche38
    on May 24, 2022

    One workaround currently is to not use _asyncio speed up module but to use pure python version as that does not has this limitation.

    interesting. How to you prevent using _asyncio ?

  11. kumaraditya303 commented on May 24, 2022

    @kumaraditya303
    kumaraditya303
    on May 24, 2022
    Contributor

    Before importing asyncio add this line:

    import sys
sys.modules["_asyncio"] = None

  12. pipiche38 commented on May 24, 2022

    @pipiche38
    pipiche38
    on May 24, 2022

    @kumaraditya303 if ayncio is imported in many module. does it enough to do it only once while importing asyncio for the first time ?

    [edit] at least I did it prior the first occurence of import and this looks pretty good .

  13. 24 remaining items

  14. kumaraditya303
    closed this as completedon Nov 29, 2022
  15. Repository owner moved this from In Progress to Done in Subinterpreterson Nov 29, 2022
  16. Repository owner moved this from In Progress to Done in asyncioon Nov 29, 2022

  17. ericsnowcurrently commented on Nov 29, 2022

    @ericsnowcurrently
    ericsnowcurrently
    on Nov 29, 2022
    Member

    Thanks for working on this, @kumaraditya303!

  18. erlend-aasland commented on Jan 23, 2023

    @erlend-aasland
    erlend-aasland
    on Jan 23, 2023
    Contributor

    Reopening: the future iter object freelist is still a static global, and we forgot to clean up Tools/c-analyzer/cpython/globals-to-fix.tsv.

  19. erlend-aasland
    reopened this on Jan 23, 2023
  20. github-project-automation
    moved this from Done to In Progress in asyncioon Jan 23, 2023
  21. erlend-aasland
    added a commit that references this issue on Jan 23, 2023

    pythonGH-91375: Purge asyncio variables from globals-to-fix list

    ebe9877

  22. kumaraditya303 commented on Jan 24, 2023

    @kumaraditya303
    kumaraditya303
    on Jan 24, 2023
    Contributor

    Reopening: the future iter object freelist is still a static global, and we forgot to clean up Tools/c-analyzer/cpython/globals-to-fix.tsv.

    I left that freelist intentionally as it is not an issue until we have per interpreter GIL. @markshannon has ideas for a global better freelist so hopefully we should be able to remove this freelist entirely.

  23. erlend-aasland commented on Jan 24, 2023

    @erlend-aasland
    erlend-aasland
    on Jan 24, 2023
    Contributor

    I left that freelist intentionally as it is not an issue until we have per interpreter GIL. @markshannon has ideas for a global better freelist so hopefully we should be able to remove this freelist entirely.

    I see, thanks for the heads-up.

  24. erlend-aasland
    added a commit that references this issue on Jan 24, 2023

    pythonGH-91375: Purge asyncio variables from globals-to-fix list

    1992b71
  25. kumaraditya303
    added a commit that references this issue on Jan 24, 2023

    GH-91375: Purge `asyncio` static variables from globals-to-fix list (#…

    00d092c
  26. erlend-aasland
    closed this as completedon Jan 24, 2023
  27. github-project-automation
    moved this from In Progress to Done in asyncioon Jan 24, 2023

  28. gvanrossum commented on Jan 24, 2023

    @gvanrossum
    gvanrossum
    on Jan 24, 2023
    Member

    I left that freelist intentionally as it is not an issue until we have per interpreter GIL. @markshannon has ideas for a global better freelist so hopefully we should be able to remove this freelist entirely.

    Hm. I wouldn't necessarily count on Mark's single freelist idea to be implemented before we get a per-subinterpreter GIL (esp. since the work on mimalloc seems stalled, alas -- Christian Heimes seems to be occupied by other things).

    So I think it would behoove us to move fi_freelist and fi_freelist_len to the "per runtime" structure? Or what am I missing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.12only security fixestopic-asynciotopic-subinterpreterstype-bugAn unexpected behavior, bug, or error

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions