create_task() recommendation seems like bad practice

Documentation

My understanding of asyncio is that all tasks etc. should be awaited on. If this is not done, then exceptions in tasks will never be seen and asyncio will give an "exception was not retrieved" warning.

The documentation currently recommends this code for a fire-and-forget task creation:

background_tasks = set()
for i in range(10):
    task = asyncio.create_task(some_coro(param=i))
    background_tasks.add(task)
    task.add_done_callback(background_tasks.discard)

https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task

This seems like bad practice as the tasks are never awaited (another problem might be if this is copied inside a function, won't the set get garbage collected after the function ends?). To my knowledge, there is no trivial way to do fire-and-forget tasks using stdlib and I have always recommended users to try the aiojobs library to handle this. e.g. In the aiohttp documentation:
https://docs.aiohttp.org/en/latest/web_advanced.html#web-handler-cancellation
(Scroll down to the second warning, and the paragraph preceding it).

I feel that this example should probably be scrapped and possibly a recommendation to use aiojobs for these tasks may be better.

I'd also suggest including the text of both error messages ("exception not retrieved" and "Task was destroyed"), so that people putting that error into a search engine are likely to end up on this page.

Linked PRs

I think there's a misunderstanding here about what create_task does. Which is understandable: the name create_task is arguably misleading by omission. It isn't just a pure function that creates and returns a Task instance; it creates a task and schedules it to the event loop. (See the documentation.)

This means the coroutine doesn't need to be explicitly awaited, because the event loop will take care of ensuring that it gets waited on. ~~The event loop also keeps a reference to the Task object, which keeps a reference to the coroutine, so nothing will be garbage collected until it's done running.~~ (Edit: this is wrong.)

So this documented recommendation for fire-and-forget task creation does work fine, and is not bad practice (except inasmuch as TaskGroups are probably a better option in most cases.)

This means the coroutine doesn't need to be explicitly awaited, because the event loop will take care of ensuring that it gets waited on.

No, it ensures it executes, not that it get awaited on. You can't get an "exception not retrieved" if the code hasn't run to create an exception...

Simple reproducer:

>>> import asyncio
>>> async def foo():
...  raise RuntimeError()
... 
>>> async def main():
...  asyncio.create_task(foo())
...  await asyncio.sleep(1)
... 
>>> asyncio.run(main())
Task exception was never retrieved
 ...

Well written code should never see "Task exception was never retrieved".

So this documented recommendation for fire-and-forget task creation does work fine, and is not bad practice (except inasmuch as TaskGroups are probably a better option in most cases.)

TaskGroups can only be used for this if you create some kind of global TaskGroup for your application that you keep around till the application ends. This can work, but in something like a server application, I think you'd only get the exceptions when the server shuts down, which could be weeks after the exception happened. So, for server applications, it'd still be best to use aiojobs or similar.

Sorry, I did misunderstand your report.

The name "fire-and-forget" specifically suggests that you don't care if it succeeds or fails. If you are going to check the result/exception, then it isn't really fire and forget.

I'm not sure if we should be recommending a specific third-party library here, but it may make sense to add a note in the documentation that fire-and-forget means you won't know if it succeeded or failed, and you may get unretrieved exception warnings in case of failure.

I'll reopen this so asyncio experts can weigh in on what, if anything, should be improved here.

Yeah, I'm not too sure about linking to a third-party library either, I just don't have a better suggestion (although TaskGroup can certainly be recommended in some circumstances).

Good engineering practice is not to ignore all exceptions though, so I'm sceptical that was the intended design, and not just something that was overlooked. Typically, a user wants to run tasks in the background without waiting for it to complete or needing to retrieve the result. That doesn't mean they don't want to know when they've broken their code and it's not running anything...

The coroutine running in the background task could contain a try/except clause that catches all exceptions and logs them. You could easily make a decorator that does this for your application so you don't have to repeat the try/except/log boilerplate in every background task you write.

We're working on additional flexibility for TaskGroup that will make this pattern simpler. (See e.g. https://discuss.python.org/t/revisiting-persistenttaskgroup-with-kotlins-supervisorscope/18417)

We're working on additional flexibility for TaskGroup that will make this pattern simpler. (See e.g. https://discuss.python.org/t/revisiting-persistenttaskgroup-with-kotlins-supervisorscope/18417)

Right, I think if TaskGroup starts supporting logging exceptions as they occur, then it should be able to replace aiojobs.Scheduler.

So, I think we should be recommending a long-lived TaskGroup for fire-and-forget tasks then, instead of the currently documented workaround.

It might also be worth having a TaskGroup.close() (and .start()?) to support this use case too. e.g. In startup/shutdown hooks:

async def on_startup(app):
    app["tg"] = asyncio.TaskGroup()
async def on_shutdown(app):
    await app["tg"].close()

In aiohttp, we have cleanup_ctx, where we'd use async with, but I'm sure there are frameworks without that kind of hook.

(Edit: I see discussion of a .shutdown() in that conversation, so maybe that's already coming).

TaskGroup itself won't be logging exceptions, you will have to write your own exception handler or done-callback for that.

For TaskGroup.start() and .close() you can call __aenter__() and __aexit__() directly (these are public APIs).

I don't see an action item here.

TaskGroup itself won't be logging exceptions, you will have to write your own exception handler or done-callback for that.

But, it will default to asyncio.get_running_loop().call_exception_handler(), right? That's the same as aiojobs, so it should still work as a simple replacement.

For TaskGroup.start() and .close() you can call __aenter__() and __aexit__() directly (these are public APIs).

Fair enough, I think generally people discourage use of calling magic methods directly, so I was thinking users will be more comfortable with a public (I mean, no _ prefix) method. I think that'd also be consistent with other objects too (e.g. file-like objects have .close() in addition to __exit__()). Either way, it'll work though.

~~> But, it will default to asyncio.get_running_loop().call_exception_handler(), right? That's the same as aiojobs, so it should still work as a simple replacement.~~

I would like to push back on this a little. Why do you want it to call call_exception_handler()? Looking over the asyncio source code, that function is generally only called when there is no alternative, e.g. when an exception is caught in a finalizer.

~~The regTaskGroup doesn't call it when a task crashes~~

[Submitted by mistake, see full comment below.]

I've been busy for closing up my company's seris A funding along with several big customer follow-ups. 😞 Recently, I began to look into this issue again, with some rewriting of the aiotools experimentation to make the key concepts in TaskGroup/Supervisor reusable to write various coroutine aggregation patterns.

achimnol/aiotools#58

I agree with that "a user should not end up spending hours or days trying to figure out why their code is not behaving correctly, simply because they never even knew an exception was happening in their code." because I also experienced this many times. 😭

Moving this issue status back to To Do as I do not believe that anyone is currently documenting this. But it may be helpful for someone interested in Docs to do so.

+1 for anything that avoids "Future exception was never retrieved".

I was trying to implement a TaskGroup that would have similar behavior to asyncio.gather(*task_tuple, return_exceptions=True). I figured aiotools.PersistentTaskGroup was a good place to start, but the unawaited tasks were very annoying to deal with. The exception handlers in aiojobs.Scheduler and aiotools.Supervisor also behave very strangely. I'm not sure how they're meant to be used.

Bunch of examples here: #101581 (comment)

The exception handlers in aiojobs.Scheduler and aiotools.Supervisor also behave very strangely. I'm not sure how they're meant to be used.

I'm not too sure what the complication is here? I think the proposed solution is likely to work very similarly. Atleast for aiojobs, which I'm more familiar with, the default behaviour is to call asyncio.AbstractEventLoop.call_exception_handler(), which if not configured to do something else, will just log the exception to stderr. A callable can also be passed in order to use your own custom function instead of using call_exception_handler() if desired.

Maybe I'm just really misinterpreting something, but I can't get the exception_handler to run at all.

This was my attempt at using it to populate an ExceptionGroup.

import asyncio
from typing import Any

import aiojobs

MULTIPLE_EXCEPTIONS_MESSAGE = "Multiple exceptions occurred"


async def get_string() -> str:
    return "hello"


async def get_int() -> int:
    return 1


# async def raise_exception() -> typing.Never: # also works
async def raise_exception(s: str, *, fail: bool = True) -> None:
    if fail:
        raise Exception(s)


async def slow_print(s: str) -> None:
    await asyncio.sleep(1)
    print(s)


async def test_scheduler(
    *,
    fail: bool = True,
) -> tuple[str, int, None, None]:
    exceptions = []

    def exception_handler(
        scheduler: "aiojobs.Scheduler",
        context: dict[str, Any],
    ) -> None:
        print("never happens")
        exceptions.append(context["exception"])

    async with aiojobs.Scheduler(exception_handler=exception_handler) as scheduler:
        t1 = await scheduler.spawn(get_string())
        t2 = await scheduler.spawn(get_int())
        t3 = await scheduler.spawn(raise_exception("Raise Exception", fail=fail))
        t4 = await scheduler.spawn(slow_print("Slow Print"))
        await scheduler.spawn(raise_exception("Raise Exception 2", fail=fail))

    if exceptions:
        raise BaseExceptionGroup(
            MULTIPLE_EXCEPTIONS_MESSAGE,
            exceptions,
        )

    await t3.wait()  # Raises exception
    return await t1.wait(), await t2.wait(), None, None

Maybe I'm just really misinterpreting something, but I can't get the exception_handler to run at all.

That's because they are thrown during the close cycle when exiting the context manager, which seems to result in suppressing the exceptions. Not sure that makes sense, so we should probably change that behaviour. But for now, if you add a sleep before exiting the context, then it does work as expected.

Not sure that makes sense, so we should probably change that behaviour.

Definitely a bug I introduced when added the wait_and_close() behaviour: aio-libs/aiojobs#546

gh-104091: Suggest TaskGroup for running background tasks in asyncio.…

[3.13] gh-104091: Suggest TaskGroup for running background tasks in a…

[3.14] gh-104091: Suggest TaskGroup for running background tasks in a…

[3.15] gh-104091: Suggest TaskGroup for running background tasks in a…

pythongh-104091: Suggest TaskGroup for running background tasks in as…

Navigation Menu

Repository navigation

create_task() recommendation seems like bad practice #104091

Documentation

Linked PRs

9 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

create_task() recommendation seems like bad practice #104091

Description

Documentation

Linked PRs

Activity

carljm commented on May 2, 2023

Dreamsorcerer commented on May 2, 2023

carljm commented on May 2, 2023

Dreamsorcerer commented on May 2, 2023

gvanrossum commented on May 2, 2023

Dreamsorcerer commented on May 2, 2023

gvanrossum commented on May 3, 2023

Dreamsorcerer commented on May 3, 2023

gvanrossum commented on May 9, 2023

9 remaining items

achimnol commented on Jul 9, 2023

willingc commented on Jun 19, 2024

Jonathan-Landeed commented on Mar 25, 2025

Dreamsorcerer commented on Mar 25, 2025

Jonathan-Landeed commented on Mar 25, 2025

Dreamsorcerer commented on Mar 25, 2025

Dreamsorcerer commented on Mar 25, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions