Introducing try..catch

If we find we really need new syntax to flag that people have thought about the consequences of multi-exceptions, we could switch the existing try..except syntax to try..catch. It seems most other languages use the latter keyword anyway.

Syntax

The syntax would just replace except with catch, leaving everything else the same.

We have the option however of disallowing catch: (i.e. with no exception, the catch-all block) -- forcing people (and automatic translators) to write catch BaseException:.

Note that catch would have to be a soft keyword (supported by the new PEG parser, see PEP 617), since there are plenty of other uses of catch in existing code that we don't want to break.

Transition

The transition plan would be that try..exept will eventually be removed from the language. There would be three stages:

try..catch and try..except can both be used.
try..except works but gives a deprecation warning.
try..except stops working.

Possibly stage 2 can be split and try..except inside async functions can be deprecated sooner than in other contexts.

During stages 1 and 2, each try statement must use either catch or except -- you cannot have both catch and except blocks in the same statement. (But you can have them in the same file.)

Semantics

When the raised exception is not a multi-exception the semantics of try..catch is the same as for try..except.

When the raised exception is a multi-error the semantics change.

Basically when a multi-error contains different exception types it is possible that more than one catch block runs. E.g.

try:
    raise MultiError([ValueError(), ZeroDivisionError(), RuntimeError()])  # or whatever
catch ValueError:
    print("VE")
catch RuntimeError:
    print("RE")

would print "VE" and "RE" and then raise (bubble up) RuntimeError() (or MultiError([RuntimeError()]).

If there's an else block it only gets run if no exceptions were raised in the first place.

If there's a finally block it gets run after all catch blocks (if any) have run, before bubbling up the unhandled exceptions (if any).

The order in which the exceptions in the multi-error are handled is just the order in which the MultiError object regurgitates them.

Multiple exceptions of the same type

This is an open issue.

What if the try block raises MultiError([ValueError("A"), ValueError("B")])? We could define different semantics.

Note that there would also be a question about two different exception classes that derive from the same base class, where the catch class specifies that base (or ultimately catch BaseException). So we cannot rely on MultiError to split exceptions based on class before we start matching exceptions to catch blocks.

TO BE CONTINUED IN A LATER COMMENT (I deleted some meta-comments related to this.)

@gvanrossum

I very much like the semantics and look of the proposed try...catch! +1 from me (based on what's currently written).

My only hesitation really is about selling the deprecation and eventual removal of try...except in the PEP; considering the amount of existing non-concurrent code, docs, books, etc. that already uses it (where there would be no benefit from switching to try...catch). Assuming there's zero risk of that code ever seeing a multi-exception, they could automatically switch everything to using catch, but for stateful code that relies on an operation occurring once (such as a rollback), propagating a multi-exception to it could be rather disastrous (I believe @gvanrossum brought this up during the sprint meeting as an argument against changing the behavior of try...except).

Also, I anticipate that if except were to be eventually removed, the number of versions between the deprecation and removal would have to be much higher than the usual (maybe even 4.x?).

During the phase that except is around and multi-exceptions exist, would a try...except simply ignore it and treat the entire exception group as if it were uncaught or instead somehow direct users towards using try...catch with a more specific error? The latter seems preferable to me, although it could come at a slight cost to existing code that uses catch (the main concern being frequently executed functions with try: d['k'] except KeyError: ....

What if the try block raises MultiError([ValueError("A"), ValueError("B")])? We could define different semantics.

I'd expect that if there was a catch ValueError block, it would be run twice for both of the above exceptions, or "swallow" both at once and execute a single time with catch Group[ValueError]. If a user wants to handle the passed arguments differently for "A" and "B", they could access the .args of the exception object (this is assuming that they could use catch ValueError as ex, and on the first execution ex would be ValueError("A") and on the second ex would be ValueError("B")). To me, it makes sense based on the other logic of catch, but this might be too implicit.

As an alternative, maybe we could use something along the lines of the pattern matching syntax to catch an exception based on its arguments. However, that does come with the potential issue of causing breakages when exception messages change (which is not something that should be currently relied on since they're not guaranteed to be the same across versions).

Note that there would also be a question about two different exception classes that derive from the same base class, where the catch class specifies that base (or ultimately catch BaseException). So we cannot rely on MultiError to split exceptions based on class before we start matching exceptions to catch blocks.

This does not solve the issue, but I think part of this could be simplified if we were to specifically forbid catch BaseException. In cases like asyncio that rely on CancelledError for signaling (which inherits from BaseException), catch BaseException would definitely be a harmful anti-pattern.

Regarding catch semantics for multiple exceptions of the same type - we could do this:

try:
    raise ValueError(1), ValueError(2)
catch ValueError as e:
    # this block executes twice, once for each ValueError

try:
    raise ValueError(1), ValueError(2)
catch Group[ValueError] as e:       # or *ValueError or some other variation
    # this block executes once, for e which is the whole group (tuple or something)

And this is a parser error:

try:
    raise ValueError(1), ValueError(2)
catch Group[ValueError] as e:       # or *ValueError or some other variation
    pass
catch ValueError as e:
    pass

Regarding catch semantics for multiple exceptions of the same type - we could do this:

It looks like at least the two of us agree on the semantics of multiple exceptions of the same type -- those examples do a good job of effectively expressing what I wrote in my above comment (other than the parser error, but I agree with that as well). :-)

Actually currently you can do this:

>>> try:
...     pass
... except ValueError as e:
...     pass
... except ValueError as e:
...     pass
...
>>>

I guess the first matching block wins. So maybe we don't need to trouble the parser with this.

I guess the first matching block wins. So maybe we don't need to trouble the parser with this.

Although that's the current behavior, I'm not sure that it would be at all desirable, and might be a bit more of a subtle issue with there being a catch Group[ValueError] and a catch ValueError immediately after. Over on the pattern matching side of things (PEP 634), during the sprint, there was work on the following causing a syntax error:

>>> match ...:
...     case spam | eggs if whatever():
...         pass
... 
  File "<stdin>", line 2
SyntaxError: the previous pattern always matches, making this one unreachable

I think a somewhat similar approach could be taken with exception groups, and would likely help users that are just learning how catch works, with it having more complexity than except.

Multiple exceptions of the same type

What if the try block raises MultiError([ValueError("A"), ValueError("B")])? We could define different semantics.

Note that there would also be a question about two different exception classes that derive from the same base class, where the catch class specifies that base (or ultimately catch BaseException). So we probably can't rely on MultiError to split exceptions based on class before we start matching exceptions to catch blocks.

Another concern here is that static type checkers expect the type of e in except ValueError as e: to be ValueError, and we should keep this for catch ValueError as e:. We can preserve this by running the catch block multiple times, once for each matching exception.

Another approach might be to allow a * in the syntax, like this:

catch *ValueError as e:  # type of e is MultiError[ValueError]

However I don't know how this would work if a superclass of some exception type is being caught:

class E(Exception): ...
class E1(E): ...
class E2(E): ...

try:
    raise MultiError([E1(), ValueError(), E2()])
catch *E as e:
    # Ideally e would be MultiError([E1(), E2()])

To unpack this problem I need to take a few steps back.

=== Translation into pseudo bytecode

Let's consider how try/except is currently translated into bytecode. Consider:

try:
    foo()
except E as e:
    bar(e)

This currently translates to something like

    SETUP_FINALLY (handler1)
    <foo()>
    POP_BLOCK
    JUMP_FORWARD (done)

handler1:
    DUP_TOP
    <load E>
    JUMP_IF_NOT_EXC_MATCH(cleanup)
    POP_TOP
    STORE_FAST(e)
    POP_TOP
    SETUP_FINALLY(handler1a)
    <bar(e)>
    POP_BLOCK
    POP_EXCEPT
    <e = None; del e>
    JUMP_FORWARD(done)
handler1a:
    <e = None; del e>
    RERAISE
cleanup:
    RERAISE

done:

(This looks differently from when I first invented Python bytecode, though not terribly so. :-)

Let's try to translate it to Python code. We get something like

try:
    foo()
except BaseException as _err:
    if _exc_match(_err, E1):
        e = _err
        bar(e)
        <dance to erase e>
    elif _exc_match(_err, E2):
        e = _err
        <other handler code>
        <dance to erase e>
    else:
        raise

Except that the <dance to erase e> stuff actually looks more like this:

    try:
        e = _err
        <handler code>
    finally:
        e = None
        del e

So if we factor that out, our rewrite looks a little simpler, even if we add a second except block:

try:
    <try block>
except BaseException as _err:
    if _exc_match(_err, E1):
        <handler 1>
    elif _exc_match(_err, E2):
        <handler 2>
    else:
        raise

Now I think we have the tools in hand to propose rewrites for multi-errors.

Handling multi-errors this way

The semantics that call the handler block once for each exception it matches would be like this, modulo several details:

try:
    <try block>
except MultiError as _multi:
    unhandled = []
    for _err in _multi:
        if _exc_match(_err, E1):
            <handler 1>
        elif _exc_match(_err, E2):
            <handler 2>
        else:
            unhandled.append(_err)
    if unhandled:
        raise MultiError(unhandled)

But each <handler> block should have its own try..except to deal with errors bubbling out of there, something like this:

try:
    <user handler code>
except BaseException as _err1:
    unhandled.append(_err1)

(It's really a bit more complicated still due to exception context for bare raise and raise..from, leaving this as an exercise for the reader.)

There's also another complication, the additional structure inside MultiError (or, in Trio's terms, ExceptionGroup) used for tracebacks that share a common tail. We need to preserve this structure, so instead of just calling MultiError(unhandled) we probably will have to call something like _multi.extended_subset(unhandled) (or perhaps exceptions raised from handlers will have to be collected in a separate list that is passed in separately -- though even those will have to keep their __cause__ and __context__ attributes).

Anyway, these added complexities seem manageable.

(Another issue is how to handle non-multi-errors without adding more code. Another exercise for the reader.)

But now for the alternative proposal, using similar primitives:

Handling multi-errors using `catch *E as e:`

Maybe this requires us to mutate _multi in place, something like this:

...
except MultiError as _multi:
    # No more loop; no elif either
    if (e := _multi.exc_match(E1)) is not None:
        <handle E1>
    if (e := _multi.exc_match(E2)) is not None:
        <handle E2>
    if not _multi.is_empty():
        raise _multi

Here _multi.exc_match(E) needs to extract all exceptions that match E (including subclasses) -- it basically splits the original MultiError into two subsets, one of individual exceptions that match E, one of individual exceptions that don't match E. It returns the matching exceptions as another MultiError instance, and prunes the original down to the non-matching exceptions.

Again the handlers need to somehow push exceptions that bubble out of them back into _multi, something like

try:
    <user handler code>
except BaseException as _err1:
    _multi.push_back(_err1)

This should of course take context, cause and bare raise (re-raise) into account.

Honestly, both approaches look doable, though maybe it wouldn't be great to combine catch E and catch *E in the same try statement.

Another option

Maybe we needn't switch from except to catch -- maybe the new syntax can just be except *E as e: and in that case the type of e will be MultiError[E]. Either all handlers should use *E or none of them should.

The semantics for except E should then be entirely unchanged from the current semantics (in particular, at most one handler runs), and if a multi-error is ever raised from the try block in that case, it will not be caught except by except BaseException and by bare except: (and by except MultiError).

The alternative at the end ("Another Option") means that as the user of an async library, if you raise X you must catch *X. If you try to catch what you raised you won't get it. I think this will be confusing.

class MultiTraceback(Traceback):
    tb_frame : Frame
    tb_next_all: Dict[BaseException, Traceback]

    def __init__(self, frame, tb_next_all = {}):
        self.tb_frame = frame
        self.tb_next_all = tb_next_all

    def add(exc : BaseException):
        ''' add an exception to this tb group '''
        tb_next_all[exc] = exc.tb

    def split(List[BaseException] excs):  
        ''' remove excs from this tb group and return a 
        new tb group for them, with same frame
        '''
        r = {(k,v) for k,v in self.tb_next_all if k in excs}
        self.tb_next_all.remove(r)
        return MultiTraceback(self.tb_frame, r)

class MultiException(BaseException):
    excs: List[BaseException]
    tb : MultiTraceBack

    def __init__(self, excs: List[BaseException], tb=None):
        self.excs = excs
        if tb:
            self.tb = tb
        else:
            self.tb = MultiTraceback(currentFrame())
            for e in excs:
                 self.tb.add(e)

    def add_exc(self, e):
        self.excs.add(e)
        self.tb.add(e)

    def exc_match(self, E):
        ''' remove the exceptions that match E
        and return them in a new MultiException
        '''
        matches = set()
        for e in self.excs:
            if match(e, E):                
                matches.add(e)
        self.excs.remove(matches)
        tb = self.tb.split(matches)
        return MultiException(matches, tb)

    def push_frame(self, frame):
        self.tb = Traceback(frame, tb_next=self.tb)

@gvanrossum

Now we use it in @gvanrossum's code snippets like this:

    if (e := _multi.exc_match(E1)) is not None:
        tb1 = _multi.tb.split(e)    #  <----     remove the matched exceptions from the traceback
        <handle E1>

At the end of all the except clauses:
_multi contains the unhandled exceptions, so we do

[ _multi.add_exc(e) for e in new_exceptions_we_raised_here]

and raise _multi

Thanks! We should probably compare this to python-trio/trio#611

Perhaps OT, but here's a list of things (probably incomplete) in the life of an exception.

Constructed. (No traceback attached.)
Raised. (New traceback item pushed.)
Bubbled out of a function into the calling frame. (New traceback item pushed.)
Caught.
Re-raised by bare raise. (No traceback pushed.)
Raised again by raise e. (New traceback item pushed.)
Raised by raise e inside an except block. (New traceback item pushed, and __context__ set to the exception being handled.)

In all cases, "traceback item pushed" is basically

e.__traceback__ = traceback(tb_frame=sys._getframe(), tb_next=e.__traceback__)

Also, raise e from e1 sets e.__cause__ = e1 and otherwise does whatever raise e does. (In particular, doing this inside an except block sets both __cause__ and __context__.)

There's also raise e.with_traceback()

Oh! So

raise e.with_traceback(tb)

seems to be equivalent to

e.__traceback__ = tb
raise e

Should there be an option to break - as in, if I got this type of exception then I don't want to process any of the others?

I don't know how common that use case would be -- they can solve it with a flag the set and test in the other handlers.

break/continue can always be added later if we decide to.

break/continue can always be added later if we decide to.

No, they can't be - there might be a break in an except block where the whole try-except is inside a loop.

I've been thinking about how task groups and exceptions groups would be used by asyncio users and independently arrived to what Guido proposed in #3 (comment) and in #3 (comment).

Below is my thought train that led me to it. Apologies for the long write up, but I hope this will help us.

Types of errors

Fundamentally there are two kinds of exceptions: control flow exceptions and operation exceptions. The examples of former are KeyboardInterrupt, asyncio.CancelledError, etc. Latter are TypeError, KeyError, etc.

When writing async/await code that uses a concept of TaskGroups (or Trio's nurseries) to schedule different code concurrently, the users should approach these kinds in a fundamentally different way.

Operation exceptions such as KeyError should be handled within the async Task that runs the code. E.g. this is what users should do:

try:
  dct[key]
except KeyError:
  # handle the exception

and this is what they shouldn't do:

try:
  async with asyncio.TaskGroup() as g:
    g.create_task(task1); g.create_task(task2)
except *KeyError:
  # handling KeyError here is meaningless, there's
  # no context to do anything with it but to log it.

Control flow exceptions are a different beast. If, for example, we want to cancel an asyncio Task that spawned multiple concurrent Tasks in it with a TaskGroup, we need to make sure that:

CancelledErrors will be propagated to all still running tasks within the group
CancelledErrors will be propagated to the Task that scheduled the group
CancelledErrors will be propagated to the outer Tasks until either the entire program stops, or the exception is handled and silenced (e.g. by asyncio.wait_for()).

So suppose we have the except *ExceptionType syntax that can only handle an ExceptionGroup with ExceptionType in it. This means that we'd see a lot of code like this:

try:
  async with asyncio.TaskGroup() as g:
    g.create_task(task1); g.create_task(task2)
except *CancelledError:
  log('cancelling server bootstrap')
  await server.stop()
  raise
except CancelledError:
  # Same code, really.
  log('cancelling server bootstrap')
  await server.stop()
  raise

Which led me to the conclusion that except *CancelledError as e should both:

catch an individual standalone CancelledError, wrap it in an ExceptionGroup as bind it to e. The type of e is always ExceptionGroup[CancelledError].
if an exception group is propagating through the try, except *CancelledError should split the group and handle all exceptions at once with one run of the code in except *CancelledError.

Why "handle all exceptions at once with one run of the code code in except *"? Why not run the code in the except clause for every matching exception that we have in the group? Basically because there's no need to. As I mentioned above, catching operation exceptions should be done with the regular old except KeyError within the Task boundary, where there's context to handle that KeyError. Catching control flow exceptions is needed to react to some global signal, do cleanup or logging, but ultimately to either stop the signal or propagate it up the caller chain.

Separating exceptions kinds to two distinct groups (operation & control flow) leads to another conclusion: an individual try..except block usually handles either the former or the latter, but not a mix of both. Which led me to conclusion that except *CancelledError should switch the behavior of the entire try block to make it run several of its except * clauses if necessary. So:

try:
  # code
except KeyError:
  # handle
except ValueError:
  # handle

is the old and familiar try..except, we don't need to change it. And:

try:
  # code
except *TimeoutError:
  # handle
except *CancelledError:
  # handle

is an entirely different mode and it's OK, and moreover, almost expected from the user standpoint, to run both except clauses here.

And:

try:
  # code
except ValueError:
  # handle
except *CancelledError:
  # handle

is weird and most likely suggests that the code should be refactored.

Types of user code

Fundamentally we have applications and libraries. Applications are somewhat simpler -- they typically can dictate what version of Python they require. Which makes introducing TaskGroups and the new except * clause somewhat easier. Basically, upon switching to Python 3.10, the application developer can grep their application code for every control flow exception they handle (search for except CancelledError) and mechanically change it to except *CancelledError. Generally, judging by my own experience, there are not so many places with code like that. Typically things like CancelledError and TimeoutError are handled only in a few places, the rest of the code relies on try..finally to cleanup its state.

Library developers are in a worse position: they'll need to maintain backwards compatibility with older Pythons, so they can't start using the new except * syntax. Two thoughts here:

This means that we'll need to have a proper programmatic API to work with ExceptionGroups, so that libraries can write code like:

try:
  # run user code
except CancelledError:
  # handle cancellation
except ExceptionGroup as e:
  g1, g2 = e.split(CancelledError)
  # handle cancellation

The API isn't going to be pretty, but that's OK, because a lot of existing asyncio libraries don't run user-provided coroutines that might use TaskGroups. In other words, a mysql or redis driver will never be able to catch an ExceptionGroup until it starts using TaskGroups itself.

Summary

I just cannot wrap my head around introducing try..catch syntax to Python. I don't think we actually need it, with the right approach to documentation we can explain to the users how to use TaskGroups and except * syntax correctly and I believe it will only make their code better. Therefore, to conclude I think we should:

Introduce the ExceptionGroup object. Tweak the interpreter to recognize them and do the correct thing to the wrapped Tracebacks etc.
ExceptionGroups should have an API so that the few libraries that run user code can maintain compatibility with older Pythons.
except *CancelledError as e should catch:
- an individual CancelledError; it would be wrapped in an ExceptionGroup created ad-hoc.
- a single CancelledError wrapped in an ExceptionGroup
- multiple CancelledErrors at once.
If a try block has two or more except * clauses it can run them one after another.
Perhaps, if there's one except * in a try block, we should require all other except clauses to be written as except *.
Handling exceptions like *TimeoutError is rather pointless. It's hard to correlate what specific subset of child tasks timed out -- the user should really handle timeouts in the sub-tasks where you have the context. So IMO exceptions groups are all about handling global signals like CancelledError or KeyboardInterrupt and about preserving the clear history of what crashed where and how to simplify development, debugging, and diagnostics.

break/continue can always be added later if we decide to.

No, they can't be - there might be a break in an except block where the whole try-except is inside a loop.

Oh, any flow control other than raise should probably just interrupt the flow -- if handler1(e) says break (it's not really a function, just the code from the block) then none of the other stuff happens except finally.

But we need to think more about this later.

Adding to #3 (comment):

Perhaps, if there's one except * in a try block, we should require all other except clauses to be written as except *.

This is also the strategy we use when designing APIs/syntax for EdgeDB: if there's no clear need to allow both except and except * clauses in one try we should only allow one of them. This is something that we can relax in later releases if we discover a use case. OTOH allowing both and later restricting it would be a breaking change.

Perhaps, if there's one except * in a try block, we should require all other except clauses to be written as except *.

Oh, definitely.

Perhaps we should open a new issue, since this one is still called "Introducing try..catch"? The new one could be called "Introducing 'except *'", and cleanly introduce the new proposal (probably linking to Irit's traceback group prototype).

The new one could be called "Introducing 'except *'", and cleanly introduce the new proposal (probably linking to Irit's traceback group prototype).

If you want I can take a stab at it.

Go for it.

Done: #4.

Initial version of the except* proposal

What's the current status of this proposal? Will this go-ahead to also soft-introduce multierrors/exception groups in the language? Or is this specific idea forgone for a different one?

A variation of this is now being proposed as PEP 654.

According to this header, this specific idea (try...catch) seems to be rejected, should this issue be closed?

Navigation Menu

Repository navigation

Introducing try..catch #3

Syntax

Transition

Semantics

Multiple exceptions of the same type

Multiple exceptions of the same type

Handling multi-errors this way

Handling multi-errors using `catch *E as e:`

Another option

16 remaining items

Types of errors

Types of user code

Summary

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Introducing try..catch #3

Description

Syntax

Transition

Semantics

Multiple exceptions of the same type

Activity

aeros commented on Oct 22, 2020

iritkatriel commented on Oct 22, 2020

aeros commented on Oct 22, 2020

iritkatriel commented on Oct 22, 2020

aeros commented on Oct 22, 2020

gvanrossum commented on Oct 22, 2020

Multiple exceptions of the same type

Handling multi-errors this way

Handling multi-errors using catch *E as e:

Another option

iritkatriel commented on Oct 22, 2020

iritkatriel commented on Oct 23, 2020

iritkatriel commented on Oct 23, 2020

iritkatriel commented on Oct 23, 2020

gvanrossum commented on Oct 23, 2020

gvanrossum commented on Oct 23, 2020

iritkatriel commented on Oct 23, 2020

gvanrossum commented on Oct 23, 2020

16 remaining items

iritkatriel commented on Oct 25, 2020

iritkatriel commented on Oct 25, 2020

1st1 commented on Oct 25, 2020

Types of errors

Types of user code

Summary

gvanrossum commented on Oct 25, 2020

1st1 commented on Oct 25, 2020

gvanrossum commented on Oct 25, 2020

gvanrossum commented on Oct 25, 2020

1st1 commented on Oct 25, 2020

gvanrossum commented on Oct 25, 2020

1st1 commented on Oct 26, 2020

ShadowJonathan commented on Feb 23, 2021

iritkatriel commented on Feb 23, 2021

ShadowJonathan commented on Feb 23, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Handling multi-errors using `catch *E as e:`