Generator .throw() exception context chain should match equivalent inline execution

[iaroslavkrasikov]

Bug report

Bug description:

Exception chaining around .throw() on generators/coroutines is inconsistent across yield from / await for native generators/coroutines and any other objects implementing Generator protocol.

Context loss when subgenerator suspends before re-raising

gh-73773 fixed exception chaining when .throw() propagates immediately. It still fails if a subgenerator suspends while handling the exception:

def inner():
    try:
        yield
    finally:
        yield
        raise

def outer():
    try:
        1 / 0
    finally:
        yield from inner()

gen = outer()
next(gen)
gen.throw(RuntimeError)
next(gen)

Traceback (most recent call last):
  ...
  File <python-input-0>, line 17, in <module>
    next(gen)
  File <python-input-0>, line 12, in outer
    yield from inner()
  File <python-input-0>, line 3, in inner
    yield
RuntimeError

Expected:

Traceback (most recent call last):
  File <python-input-0>, line 10, in outer
    1 / 0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
  File <python-input-0>, line 17, in <module>
    next(gen)
  File <python-input-0>, line 12, in outer
    yield from inner()
  File <python-input-0>, line 3, in inner
    yield
RuntimeError

The context should be attached when the exception is delivered. It should survive when inner() yields again and performs its bare raise during a later next() call.

Coroutine cleanup

asyncio users can encounter this behavior when a coroutine is cancelled while cleanup code delays the re-raise of CancelledError:

import asyncio

async def cleanup():
    try:
        await asyncio.sleep(0)
    except asyncio.CancelledError:
        await asyncio.sleep(0)  # suspension before the re-raise
        raise

async def work():
    try:
        1 / 0
    finally:
        await cleanup()

async def main():
    task = asyncio.create_task(work())
    await asyncio.sleep(0)
    task.cancel()
    await task

asyncio.run(main())

Traceback (most recent call last):
  ...
  File <python-input-0>, line 21, in main
    await task
  File <python-input-0>, line 15, in work
    await cleanup()
  File <python-input-0>, line 7, in cleanup
    await asyncio.sleep(0)
asyncio.exceptions.CancelledError

Expected:

Traceback (most recent call last):
  File <python-input-0>, line 13, in work
    1 / 0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
  File <python-input-0>, line 21, in main
    await task
  File <python-input-0>, line 15, in work
    await cleanup()
  File <python-input-0>, line 7, in cleanup
    await asyncio.sleep(0)
asyncio.exceptions.CancelledError

Nested yield from skips an intermediate context

gh-84871 reports similar issue where an intermediate exception context is lost in nested yield from calls:

def inner():
    yield

def middle():
    try:
        raise Exception(1)
    finally:
        yield from inner()

def outer():
    try:
        raise Exception(0)
    finally:
        yield from middle()

gen = outer()
next(gen)
gen.throw(Exception("thrown"))

Traceback (most recent call last):
  File <python-input-0>, line 15, in outer
    raise Exception(0)
Exception: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
  File <python-input-0>, line 21, in <module>
    gen.throw(Exception("thrown"))
Exception: thrown

Expected:

Traceback (most recent call last):
  File <python-input-0>, line 15, in outer
    raise Exception(0)
Exception: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File <python-input-0>, line 9, in middle
    raise Exception(1)
Exception: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
  File <python-input-0>, line 21, in <module>
    gen.throw(Exception("thrown"))
Exception: thrown

Context of the .throw() callsite is omitted

Consider a .throw() performed while its caller is already handling an exception:

def gen():
    yield

g = gen()
next(g)
try:
    1 / 0
finally:
    g.throw(RuntimeError)

Traceback (most recent call last):
  File <python-input-0>, line 9, in <module>
    g.throw(RuntimeError)
  File <python-input-0>, line 2, in gen
    def gen():
RuntimeError

Expected:

Traceback (most recent call last):
  File <python-input-0>, line 7, in <module>
    1 / 0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File <python-input-0>, line 9, in <module>
    g.throw(RuntimeError)
  File <python-input-0>, line 2, in gen
    yield
RuntimeError

Native and custom subgenerators disagree

The same logical code produces different chains depending on whether the subgenerator is native or implements the protocol in Python:

def native():
    try:
        raise Exception(1)
    finally:
        yield


class custom:
    def __iter__(self):
        return self

    def __next__(self):
        return None

    def throw(self, exc, *rest):
        try:
            raise Exception(1)
        finally:
            raise exc

def gen(subgen):
    yield from subgen()

g = gen(native) # or gen(custom)
next(g)
try:
    raise Exception(0)
finally:
    g.throw(Exception("thrown"))

Using native

Traceback (most recent call last):
  File "<python-input-0>", line 6, in native
    raise Exception(1)
Exception: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<python-input-0>", line 34, in <module>
    g.throw(Exception("thrown"))
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "<python-input-0>", line 26, in gen
    yield from subgen()
  File "<python-input-0>", line 8, in native
    yield
Exception: thrown

Using custom (as expected)

Traceback (most recent call last):
  File "<python-input-0>", line 32, in <module>
    raise Exception(0)
Exception: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<python-input-0>", line 20, in throw
    raise Exception(1)
Exception: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<python-input-0>", line 34, in <module>
    g.throw(Exception("thrown"))
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "<python-input-0>", line 26, in gen
    yield from subgen()
  File "<python-input-0>", line 22, in throw
    raise exc
Exception: thrown

The inlining argument

The exception chain produced by a .throw() turn should be the same as if all code executed by that turn, including code reached through yield from, await, or custom throw(), were inlined at the .throw() call site. This includes exceptions still being handled by suspended frames.

The inlined equivalent to the code above is:

try:
    raise Exception(0)
finally:
    try:
        raise Exception(1)
    finally:
        raise Exception("thrown")

Native and custom delegation should produce the same exception chain because they represent the same operation.

• Explicit chaining with raise ... from ... and __suppress_context__ should work as before.
• throw() into an exhausted native generator should work as raise at the callsite:

def gen():
    yield

g = gen()
[*g]
try:
    1 / 0
finally:
    g.throw(RuntimeError)

is equivalent to:

try:
    1 / 0
finally:
    raise RuntimeError

close() and GeneratorExit

Tracebacks with close() are broken in the similar way:

def gen():
    try:
        raise Exception(1)
    finally:
        try:
            yield
        finally:
            raise Exception(2)

g = gen()
next(g)
try:
    raise Exception(0)
finally:
    g.close()

Traceback (most recent call last):
  File "<python-input-0>", line 3, in gen
    raise Exception(1)
Exception: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<python-input-0>", line 6, in gen
    yield
GeneratorExit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<python-input-0>", line 16, in <module>
    g.close()
    ~~~~~~~^^
  File "<python-input-0>", line 8, in gen
    raise Exception(2)
Exception: 2

Exception(0) is being lost from the context. This is visible only when cleanup raises, because successful close() suppresses GeneratorExit.

Related issues

• gh-108668 reports context loss when delegated code suspends before re-raising.
• gh-111375 reports that caller’s handled exception remains visible while @contextmanager drives a generator with .throw()

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

[picnixz]

cc @iritkatriel @ZeroIntensity @kumaraditya303