[Python] Regression: grpcio 1.81.1 fails with epoll EBADF after subprocess.Popen (works in 1.78.0)

[Python] Regression: active gRPC channels crash forked subprocesses on Linux through grpcio 1.83.0

What version of gRPC and what language are you using?

Python grpcio versions 1.80.0, 1.81.1, 1.82.1, and 1.83.0. 1.78.0 is included as the passing regression control.

What operating system (Linux, Windows,...) and version?

Linux/x86_64, using the official Debian-based python:<version>-slim images. (run from arm64 macbook)

What runtime / compiler are you using (e.g. python version or version of gcc)

CPython 3.9.25, 3.10.20, and 3.12.13.

What did you do?

I started a gRPC server in a separate process, opened and warmed up a client channel, kept RPCs active in a background thread, and repeatedly started a shell subprocess using the public subprocess.Popen arguments preexec_fn and group. There are no private CPython flags, private gRPC flags, special gRPC environment variables, or generated protobuf files.

Save these two files in the same directory.

Dockerfile:

ARG PYTHON_VERSION=3.10
FROM python:${PYTHON_VERSION}-slim

ARG GRPCIO_VERSION=1.83.0
RUN python -m pip install --no-cache-dir "grpcio==${GRPCIO_VERSION}"

COPY repro.py /repro.py
ENTRYPOINT ["python", "/repro.py"]

repro.py:

#!/usr/bin/env python3

import os
import socket
import subprocess
import sys
import threading
import time
from concurrent.futures import ThreadPoolExecutor

import grpc


def run_server():
    server = grpc.server(ThreadPoolExecutor(max_workers=4))
    server.add_generic_rpc_handlers(
        (
            grpc.method_handlers_generic_handler(
                "test.TestService",
                {"Ping": grpc.unary_unary_rpc_method_handler(lambda request, _: request)},
            ),
        )
    )
    port = server.add_insecure_port("127.0.0.1:0")
    server.start()
    print(port, flush=True)
    threading.Event().wait()


if "--server" in sys.argv:
    run_server()
    raise SystemExit

print(f"Python {sys.version.split()[0]}, grpcio {grpc.__version__}", flush=True)
server = subprocess.Popen(
    [sys.executable, __file__, "--server"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    text=True,
)

try:
    port = int(server.stdout.readline())
    socket.create_connection(("127.0.0.1", port)).close()
    channel = grpc.insecure_channel(f"127.0.0.1:{port}")
    ping = channel.unary_unary("/test.TestService/Ping")
    for _ in range(20):
        ping(b"warmup", timeout=2)

    def keep_grpc_active():
        while True:
            ping(b"ping", timeout=2)
            time.sleep(0.005)

    threading.Thread(target=keep_grpc_active, daemon=True).start()

    for iteration in range(20_000):
        child = subprocess.Popen(
            ["/bin/sh", "-c", "echo out; echo err 1>&2; exit 0"],
            stdin=subprocess.DEVNULL,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True,
            close_fds=True,
            preexec_fn=lambda: None,
            group=os.getgid(),
        )
        _, stderr = child.communicate(timeout=5)
        if child.returncode:
            print(f"REPRODUCED at iteration {iteration}: rc={child.returncode}")
            print(stderr, end="")
            raise SystemExit(1)
        if iteration % 500 == 0:
            print(f"completed iteration {iteration}", flush=True)

    print("DONE: no failure observed")
finally:
    server.terminate()
    server.wait(timeout=5)

Build and run the original passing control:

docker build \
  --build-arg PYTHON_VERSION=3.9 \
  --build-arg GRPCIO_VERSION=1.78.0 \
  -t grpc-fork-repro:py39-grpc1780 .
docker run --rm grpc-fork-repro:py39-grpc1780

Build and run the latest affected release:

docker build \
  --build-arg PYTHON_VERSION=3.10 \
  --build-arg GRPCIO_VERSION=1.83.0 \
  -t grpc-fork-repro:py310-grpc1830 .
docker run --rm grpc-fork-repro:py310-grpc1830

Change PYTHON_VERSION and GRPCIO_VERSION to run the other matrix entries. The race is nondeterministic, so rerunning the container may be necessary.

What did you expect to see?

Every subprocess should exit successfully, and the reproducer should finish all 20,000 iterations with:

DONE: no failure observed

What did you see instead?

The Python 3.9/grpcio 1.78.0 control completes all 20,000 iterations. Beginning with grpcio 1.80.0, subprocesses abort while the parent has an active gRPC channel:

Python 3.9.25, grpcio 1.80.0
REPRODUCED at iteration 7136: rc=-6
E0723 18:35:36.311622 262570 ev_epoll1_linux.cc:373]
(event_engine) Epoll1Poller:0x560403a42ee0 encountered epoll_wait error:
Bad file descriptor

The latest release reproduces the same epoll_wait failure:

Python 3.12.13, grpcio 1.83.0
REPRODUCED at iteration 1199: rc=-6
E0723 18:40:57.093520 44251 ev_epoll1_linux.cc:373]
(event_engine) Epoll1Poller:0x55599fffac50 encountered epoll_wait error:
Bad file descriptor

Confirmed Linux/x86_64 matrix:

Python	grpcio	Result
3.9.25	1.78.0	PASS: 20,000 iterations
3.9.25	1.80.0	FAIL: iteration 7,136; epoll EBADF
3.10.20	1.81.1	FAIL: iteration 508; epoll EBADF
3.10.20	1.82.1	FAIL: iteration 766; epoll EBADF
3.10.20	1.83.0	FAIL: iteration 12,556; epoll EBADF
3.12.13	1.81.1	FAIL: iteration 956; epoll EBADF
3.12.13	1.82.1	FAIL: iteration 194; epoll EBADF
3.12.13	1.83.0	FAIL: iteration 1,199; epoll EBADF

Anything else we should know about your project / environment?

• This reproduces on supported Python 3.10 and 3.12, not only Python 3.9.
• preexec_fn and group are documented public subprocess.Popen arguments.
• The reproducer does not modify _USE_VFORK, _USE_POSIX_SPAWN, or any other private implementation detail.
• Different runs can produce epoll_wait ... Bad file descriptor, a failed assertion in ev_epoll1_linux.cc, or occasionally rc=-11.
• Related issue: https://github.com/grpc/grpc/issues/42062