#11323·orleans

ci(cosmos): investigate classic emulator native crash under integration load

Author: ReubenBondCreated Sep 18, 2026Updated Sep 18, 2026

Problem

The classic Linux Cosmos emulator starts with Strong consistency and serves native membership operations, then terminates inside its native SQLPAL/network stack during the full Cosmos provider suite. Subsequent membership, reminder, and persistence operations fail with connection refused. This backend process failure should be distinguished from the parallel-membership test timeout which precedes it.

Evidence

  • PR: #11312.
  • Diagnostic head: cfaced632a0a7789558cbbbc7879e48a67fc911d.
  • Run/job: https://github.com/dotnet/orleans/actions/runs/35364434859/job/105663311979 (net10.0).
  • This superseded run was cancelled at about 16:05 UTC; the native process had already crashed at 15:54:21 UTC and exited at 15:54:37 UTC. Cleanup output records the earlier engine failure.
  • Image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:2.14.28@sha256:9a4f69121f20c1d6502db84625c71ebb441ab61b01880a7b2d3d8031c6dac429.
  • Configuration: AZURE_COSMOS_EMULATOR_ARGS=/Consistency=Strong, AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=127.0.0.1, gateway access over HTTPS port 8081.
  • Native version: 2.14.28.0 (b9ee566d); evaluation period had 175 days remaining.
  • Docker state: Status=exited, ExitCode=1, OOMKilled=false.
  • Native diagnostic environment: Ubuntu 22.04.5 LTS inside the image, 4 processors, 16,766,414,848 bytes total memory.
This program has encountered a fatal error and cannot continue running
Reason: 0x00000006
Message: Last Chance
Stack Trace:
  file://package1/windows/system32/sqlpal.dll+0x000000000000A5D6
  file://package1/windows/system32/sqlpal.dll+0x0000000000008F46
  file://package1/windows/system32/sqlpal.dll+0x0000000000036B13
  ...
  file:///windows/System32/Drivers/AfdWsk.sys+0x0000000000007805
  file:///windows/System32/Drivers/Http.sys+0x000000000001E19D
  file:///windows/System32/Drivers/Http.sys+0x000000000001E138
  ...
Process: 21 - cosmosdb-emulator
Thread: 105 (application thread 0x3bc)
Crash Id: da820688-d88b-4c9e-9093-b85065d796a4

An earlier run on ec10b9a2a28e411e28e2d38c28440f55adb8a30c also lost the endpoint: https://github.com/dotnet/orleans/actions/runs/35362201727/job/105655932045. Its first-attempt TRX shows MembershipTable_Cosmos_AccountUsesStrongConsistency, MembershipTable_Cosmos_ReadRow_Insert_Read, and MembershipTable_Cosmos_UpdateRow passed. MembershipTable_Cosmos_UpdateRowInParallel timed out at the shared 30-second wait; subsequent tests encountered connection refused. That run did not yet capture the emulator's exit state/logs.

Expected outcome

A supported native Strong-consistency backend should remain available throughout both framework variants of the full Cosmos suite, including conditional transactions and parallel membership updates. Preserve real backend validation and the Strong guarantee while determining the crash trigger.

Investigation

Repository issue/PR searches for Cosmos emulator crashes, SQLPAL, and Last Chance found no existing Orleans tracking issue. Related vendor reports provide similar signatures, but a common root cause has not been established:

  • Azure/azure-cosmos-db-emulator-docker#142: an older integration-load crash with SQLPAL, AfdWsk.sys and Http.sys frames, reported as Kernel bug check.
  • Azure/azure-cosmos-db-emulator-docker#345: Last Chance / 0x00000006 termination on an earlier image with SQLPAL frames.

The native dump path was logged but its contents were not retained. Next diagnostics should preserve the native crash output and identify whether the contention timeout is a precursor, consequence, or independent issue. A Windows-native Strong emulator is a bounded CI alternative to evaluate, rather than downgrading consistency, adding skips, or treating mock results as native evidence.