[Python] grpc.aio: abandoned calls are never garbage-collected since 1.83, so never cancelled

What version of gRPC and what language are you using?

grpcio 1.84.0 (regression introduced in 1.83.0), Python

What operating system (Linux, Windows,...) and version?

Linux (x86_64, Debian trixie); also reproduced in CI containers.

What runtime / compiler are you using (e.g. python version or version of gcc)

CPython 3.14.7; also reproduced on 3.13 and 3.12.

What did you do?

A grpc.aio call that an application drops without cancelling — reading one chunk of a server stream and returning, for example — used to be cancelled by Call.__del__ once it was collected ("Cancelled upon garbage collection!"). Since 1.83.0 the call object is never collected, so the RPC is never cancelled and stays in flight until the channel closes.

"""grpcio >= 1.83: an abandoned grpc.aio call is never collected, so never cancelled."""

import asyncio
import gc
import weakref

import grpc


async def endless_stream(request, context):
    yield b"chunk-0"
    await asyncio.Event().wait()  # never terminates on its own


async def main():
    server = grpc.aio.server()
    server.add_generic_rpc_handlers(
        (
            grpc.method_handlers_generic_handler(
                "probe.Service",
                {
                    "EndlessStream": grpc.unary_stream_rpc_method_handler(
                        endless_stream, request_deserializer=None, response_serializer=None
                    )
                },
            ),
        )
    )
    port = server.add_insecure_port("127.0.0.1:0")
    await server.start()

    ref = {}
    async with grpc.aio.insecure_channel(f"127.0.0.1:{port}") as channel:

        async def read_one_chunk_then_drop_the_call():
            call = channel.unary_stream("/probe.Service/EndlessStream")(b"ping")
            ref["call"] = weakref.ref(call)
            async for _chunk in call:
                return  # abandon the call without cancelling it

        await read_one_chunk_then_drop_the_call()

        gc.collect()
        for _ in range(50):  # give __del__ and the loop every chance to run
            await asyncio.sleep(0.01)
            gc.collect()
            if ref["call"]() is None:
                break

        print("grpcio             :", grpc.__version__)
        print("call collected     :", ref["call"]() is None)
        print("RPC still in flight:", not ref["call"]().done() if ref["call"]() else False)
        await channel.close()
    await server.stop(None)


asyncio.run(main())

What did you expect to see?

grpcio             : 1.82.2
call collected     : True
RPC still in flight: False

What did you see instead?

grpcio             : 1.83.0        # and 1.84.0
call collected     : False
RPC still in flight: True

Analysis

Channel._register_call, added in 1.83.0 by #42503, registers a done callback on every call it tracks:

def _register_call(self, call: _base_call.Call) -> None:
    """Register a call to be tracked by the channel."""
    self._active_calls.add(call)
    call.add_done_callback(self._active_calls.discard)

Call.add_done_callback binds the call into the callback and hands it to the cython call:

def add_done_callback(self, callback: DoneCallbackType) -> None:
    cb = partial(callback, self)
    self._cython_call.add_done_callback(cb)

So _AioCall (C) holds a strong reference to the Python Call through that partial, while the Python Call owns _cython_call. Core keeps the cython call alive while the RPC is in flight, and the RPC is in flight until something cancels it — but the only thing that would cancel an abandoned call is Call.__del__, which cannot run while that edge exists. The call ends up unreachable but immortal, and gc.collect() cannot break the cycle because it crosses the C object.

Note that add_done_callback has always created this C→Python edge: on 1.82.2 the same reproducer also fails to collect the call if the application registers a done callback of its own. What changed in 1.83.0 is that _register_call now does it for every call, unconditionally, so the leak no longer requires the application to opt in.

Scope: which calls are affected

Only the calls that never terminate, which is why this is not immediately obvious in the wild — a call that ends fires its done callback, releases the partial, and is collected normally. Measured on 1.84.0, same channel, 25 abandoned calls:

Call outcome	Collected?
Stream consumed to completion	yes
Stream cancelled explicitly	yes
Stream abandoned without a cancel	no — 25/25 retained
…the same 25, after channel.close()	yes

So the retention is unbounded for as long as the channel lives, one object graph per abandoned stream, and a long-lived channel is the normal case for a service dialing a backend.

Impact

Each retained call holds more than memory: an in-flight RPC, whose server-side handler keeps running (in the reproducer it stays parked in await asyncio.Event().wait()), and an open HTTP/2 stream on the channel. Since in-flight streams count against MAX_CONCURRENT_STREAMS, we would expect a channel that accumulates abandoned calls to eventually stall new RPCs — we have not measured that, so it is stated as an expectation rather than an observation.

We found this as a hang rather than as memory growth: our telemetry ends a streaming client span from the call's done callback, and for an abandoned stream that callback now never fires, so the span stays open for the lifetime of the channel.

Workaround

Keep the WeakSet tracking, drop the callback:

def _register_call(self, call):
    self._active_calls.add(call)

This looks safe because _active_calls is a WeakSet — entries disappear when a call is collected, which is precisely what the callback was preventing — and _close re-reads the set, skips calls that are done() and cancels the rest, both no-ops on a completed call. A fix inside grpcio presumably wants the discard to hold only a weak reference to the call, so that tracking cannot extend its lifetime.

Anything else we should know about your project / environment?

Bisected: 1.82.2 good, 1.83.0 bad, 1.84.0 bad. Reproduces on CPython 3.12, 3.13 and 3.14.

Authorship disclosure: this report — the bisection, the reference-graph analysis, the reproducer and the measurements above — was produced by Claude Opus 5 (an AI coding agent, running in OpenCode) while investigating a CI failure in our own repository. Every result quoted here was executed against real grpcio builds in the environments listed above rather than inferred, and the reproducer runs standalone. A human reporter reviewed it before filing, but please treat the reasoning in the Analysis section as a starting hypothesis to verify rather than an authoritative diagnosis of grpcio internals.