Crash during process shutdown on Windows

What version of gRPC and what language are you using?

Language: C++

gRPC: 1.81.1
vcpkg baseline: 9e593bb18ea69cc5095e012465dcd675a822ed0d

The same behavior was also reproduced with gRPC 1.76.0.

What operating system (Linux, Windows,...) and version?

Windows 10, build 19045.6456

What runtime / compiler are you using?

Visual Studio: 2022 17.14.40
MSVC compiler: 19.44.35228
MSVC toolset: 14.44.35207
Windows SDK: 10.0.26100.0

vcpkg triplet: x64-windows-static

Configuration: Debug
C++ standard: C++20
MSVC runtime: /MTd

Relevant compiler options:

/Od /Ob0 /RTC1 /Zi

Relevant linker options:

/machine:x64 /debug /INCREMENTAL /subsystem:console

What did you do?

I created a minimal standalone C++ application that:

1. Starts a gRPC server on 127.0.0.1:0.
2. Creates a client channel to the selected port.
3. Performs one unary RPC.
4. Destroys the client channel.
5. Calls:

server->Shutdown();
server->Wait();

1. Returns from main().

A minimal reproducer is attached.

Contents of the attached issue.zip file:

main.cpp
greet.proto
greet.pb.h
greet.pb.cc
greet.grpc.pb.h
greet.grpc.pb.cc

The generated protobuf/gRPC sources are included so the reproducer can be built without regenerating them.

The crash is timing-dependent and is easiest to observe under the Visual Studio debugger.

What did you expect to see?

The process should terminate normally after all application-owned gRPC objects have been destroyed and after:

server->Shutdown();
server->Wait();

No gRPC worker thread should crash during CRT/STL shutdown.

What did you see instead?

Sometimes, during process shutdown, a gRPC worker thread crashes with:

0xC0000005: Access violation

The worker thread is typically inside MSVC Debug STL synchronization while cleaning up gRPC objects, for example:

EnterCriticalSection
_Mtxlock
std::_Lockit::_Lockit
std::_Container_base12::_Orphan_all
...
grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::ThreadBody

The specific gRPC object being destroyed may vary between runs.

At the same time, the main thread is already inside MSVC CRT shutdown, for example:

_RTC_Shutdown
_RTC_Terminate
_execute_onexit_table
common_exit
exit

At the time of the crash, gRPC cleanup is still running on a worker thread while process teardown has already started on the main thread.

Workaround and possible cause

Explicitly keeping the default EventEngine alive for the entire application lifetime and shutting it down before returning from main() makes the reproducer stable:

namespace ee = grpc_event_engine::experimental;

int main()
{
    ee::SetDefaultEventEngine(ee::CreateEventEngine());

    {
        // Create server and channel, perform RPC,
        // destroy channel, then Shutdown()/Wait() the server.
    }

    ee::ShutdownDefaultEventEngine();

    return 0;
}

With this workaround I have not been able to reproduce the crash.

Calling only:

grpc_event_engine::experimental::ShutdownDefaultEventEngine();

after the gRPC objects had already been destroyed did not reliably prevent the crash.

This suggests that the default EventEngine lifetime may be related to the problem, but this is only a hypothesis based on the observed stack traces and workaround behavior.

issue.zip