Finalization of non-exhausted asynchronous generators is deferred

@gvanrossum

BPO	44518
Nosy	@gvanrossum, @asvetlov, @markshannon, @serhiy-storchaka, @1st1, @gvanrossum

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-06-27.09:11:32.621>
labels = ['interpreter-core', '3.11', '3.9', '3.10', 'performance']
title = 'Finalization of non-exhausted asynchronous generators is deferred'
updated_at = <Date 2021-06-28.18:33:50.454>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-06-28.18:33:50.454>
actor = 'gvanrossum'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2021-06-27.09:11:32.621>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 44518
keywords = []
message_count = 6.0
messages = ['396569', '396610', '396611', '396644', '396650', '396659']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'asvetlov', 'Mark.Shannon', 'serhiy.storchaka', 'yselivanov', 'Guido.van.Rossum']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue44518'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

In the following example:

def gen():
    try:
        yield 1
    finally:
        print('finalize inner')

def func():
    try:
        for x in gen():
            break
    finally:
        print('finalize outer')

func()
print('END')

the output in CPython is:

finalize inner
finalize outer
END

But in similar example for asynchronous generator:

async def gen():
    try:
        yield 1
    finally:
        print('finalize inner')

async def func():
    try:
        async for x in gen():
            break
    finally:
        print('finalize outer')

import asyncio
asyncio.run(func())
print('END')

the output in CPython is:

finalize outer
finalize inner
END

There is a strong link somewhere which prevents finalization of the asynchronous generator object until leaving the outer function.

Tested on CPython 3.7-3.11. In PyPy "finalize inner" is not printed at all. Using closing() and aclosing() is the right way to get deterministic finalization, but in any case it would be better to get rid of strong link which prevents finalization of the asynchronous generator object.

Can you repro this without asyncio?

Sorry, I do not understand what do you mean. The problem is that non-exhausted asynchronous generators are not finalized until asyncio.run() is finished. This differs from synchronous generators which are finalized as early as possible (in CPython).

I am wondering whether the issue is in asyncio.run, or in the basic async generator implementation. If the latter, you should be able to demonstrate this with a manual "driver" (trampoline?) instead of asyncio.run.

Inlined asyncio.run(func()) is roughly equivalent to the following code:

loop = asyncio.new_event_loop()
loop.run_until_complete(func())
loop.run_until_complete(loop.shutdown_asyncgens())
loop.close()

If comment out loop.run_until_complete(loop.shutdown_asyncgens()) I get the following output:

finalize outer
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<<async_generator_athrow without __name__>()>>
END

No "finalize inner" but a warning about pending task instead.

On other hand, the following code

for _ in func().__await__(): pass

produces the output in expected order:

finalize inner
finalize outer
END

That's a rather literalist interpretation of my question. :-) I meant whether you can come up with a repro without an event loop framework, since that would presumably make reasoning about the root cause of the problem easier.

trio documents a one-tick finalization window here: https://github.com/python-trio/trio/blob/35319b3d4b881032416bfd733e0ac5239ce4c46d/trio/_core/_asyncgens.py#L26-L30 and here https://trio.readthedocs.io/en/stable/reference-core.html#finalization

Ah, sorry. The whole point of the bug seems to be that the event loop hangs on to an extra reference, and there is a reason.

I should really recuse myself from this issue since I don't understand async generators.

In PyPy "finalize inner" is not printed at all.

This looks like it could be an asyncio bug - asyncio should use the firstiter hook to keep a reference to generators that need shutting down

In PyPy "finalize inner" is not printed at all.

seems that got fixed:

import threading
import trio
import asyncio

async def gen(runner):
    try:
        yield 1
    finally:
        print(f'{runner=} finalize inner')

async def func(runner):
    try:
        async for x in gen(runner):
            break
    finally:
        print(f'{runner=} finalize outer')

print(f"{trio.run(func, 'trio')=}")
print(f"{asyncio.run(func('asyncio'))=}")
print(f"{func('manual').send(None)=}")  # on pypy3 this doesn't print finalize inner

prints:

$ pypy3.8 demo.py
runner='trio' finalize outer
runner='trio' finalize inner
trio.run(func, 'trio')=None
runner='asyncio' finalize outer
runner='asyncio' finalize inner
asyncio.run(func('asyncio'))=None
runner='manual' finalize outer
Traceback (most recent call last):
  File "demo.py", line 20, in <module>
    print(f"{func('manual').send(None)=}")
StopIteration

The behaviour cannot be changed without significant changes to the interpreter. The documentation will be handled for this as part of #100108 so closing.

For anyone who comes here for a possible fix which always works:

import contextlib

async def gen():
    try:
        yield 1
    finally:
        print('finalize inner')

async def func():
    try:
        async with contextlib.aclosing(gen()) as g:
            async for x in g:
                break
    finally:
        print('finalize outer')

import asyncio
asyncio.run(func())
print('END')

Navigation Menu

Repository navigation

Finalization of non-exhausted asynchronous generators is deferred #88684

10 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Finalization of non-exhausted asynchronous generators is deferred #88684

Description

Activity

serhiy-storchaka commented on Jun 27, 2021

gvanrossum commented on Jun 28, 2021

serhiy-storchaka commented on Jun 28, 2021

gvanrossum commented on Jun 28, 2021

serhiy-storchaka commented on Jun 28, 2021

10 remaining items

gvanrossum commented on Jul 19, 2022

graingert commented on Jul 19, 2022

gvanrossum commented on Jul 19, 2022

graingert commented on Jul 19, 2022

graingert commented on Jul 19, 2022

kumaraditya303 commented on Apr 16, 2023

kumaraditya303 commented on Apr 16, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions