We Fixed the Eval Platform: The TypeError That Took Down Three Benchmark Pipelines At 3 AM, Sentry lit up with .
Three benchmark pipelines crashed.
Not a memory leak, not a segfault, but a race condition hiding behind a TypeError, turning a high-stakes eval run into chaos.
Here is how we resolved it, with no fluff.
The Root Cause: Async Data Meets Blind Faith in .map() The error trace pointed to , where assumed would always exist.
The junior dev tested with clean data, but in production, (async) and (sync) were racing.
At 100+ RPS, was often .
The Offending Code: Why It Failed: Race Condition: was fetched asynchronously, but treated it as synchronous.
OOM Risk: Unbounded on 10K+ metrics could exhaust 8GB RAM.
Worker Starvation: No concurrency limits led to thread pool exhaustion.
The Fix: Guard Clauses, Bounded Queues, and Pragmatism Step 1: Fail Fast, Fail Loud Added zero-overhead runtime checks to reject bad data early: Why?
Stops crashes immediately.
Cost: 1-2 CPU cycles.
Negligible.
Step 2: Chunked Processing for 8GB RAM Original code processed all metrics at once, causing OOM crashes.
Fixed with 100-item chunks: Hardware Realities: 6GB Heap Limit: Leaves 2GB for the OS and other processes. : Prevents the event loop from choking.
Step 3: Bounded Worker Pool (4 Workers) Original: Unbounded concurrency caused thread pool meltdown.
Fixed with a semaphore-based pool: Usage: Why 4 Workers? 8GB RAM: 4 workers use ~2GB RAM each, with headroom for garbage collection.
CPU Bound: Matches typical 4-core cloud instances.
Step 4: Immutable Data and Network Timeouts Problem: Mutable plus async fetches led to race conditions.
Fix: Hardware Impact: 3s Timeout: Covers 99.9% of network latencies. : Zero cost.
V8 optimizes frozen objects.
Hardware Profiling: 8GB RAM, No Illusions Metric Before Fix After Fix Peak Memory (1K evals) 7.8GB (OOM crashes) 5.2GB (stable) CPU Usage (4 workers) 100% (thrashing) 60% (bounded) Error Rate 12% () 0.01% (guarded) Latency (p99) 12s (unbounded) 4s (chunked + pooled) Tuning Notes: Chunk Size: 100 items, balanced for RAM and CPU.
Worker Pool: 4 workers, matches 4-core instances.
Timeouts: 3s, because hope is not a strategy.
Failure Walkthrough: When Things Still Go Wrong Scenario 1: 10K Metrics in One Benchmark Before: OOM crash (7.8GB, OS kills it).
After: Chunked processing (100 items/chunk) caps peak memory at 5.2GB. yields the event loop, preventing starvation.
Scenario 2: Network Latency Spike (1s) Before: is , causing .
After: 3s timeout aborts stale fetch.
Immutable prevents race conditions.
Scenario 3: 200 RPS Burst Before: 200 workers exhaust the thread pool.
After: Worker pool caps at 4, bounding concurrency.
Queue backpressure enables graceful degradation.
Junior vs Senior: The Difference Between Crash and Stability Aspect Junior (Broken) Senior (Hardened) Data Handling Assumed sync Async with guards Concurrency Unbounded Bounded (4 workers) Memory OOM risk Chunked (100 items) + 6GB limit Error Handling Silent crashes Structured errors () Data Integrity Mutable state Immutable () The Bottom Line We did not reinvent the wheel.
We stopped pretending async data would magically synchronize itself.
No buzzwords, no hype, just code that does not crash under pressure.
For a template with these guardrails, see ShipMVP.
It is what we wish we had at 3 AM.
Now, tell us: what is the worst race condition you have debugged, and how did you fix it?