[BUG] Hatchet API silent crash no supervision

Describe the issue

Under a burst of individually-triggered task-creation requests, the hatchet-api process inside the hatchet-dashboard image can exit with zero log output (no panic, no fatal error, nothing). Once it exits, the container's entrypoint.sh has no mechanism to notice or restart it: nginx (which shares the container) keeps running fine and just returns 502 Bad Gateway for every request, forever. The container itself stays Up/healthy from Docker's point of view (RestartCount=0), so nothing in a typical container-level healthcheck or orchestrator would catch this either. In our test, the API stayed down for 3+ minutes until we manually ran docker compose restart hatchet-dashboard.

Environment

• SDK: Python hatchet-sdk v1.40.1
• Engine: Self-hosted, ghcr.io/hatchet-dev/hatchet/hatchet-engine:latest @ sha256:f0661c2b0245359be1bbb8f36b67a8f1387d5d3e30cfea717db5e7ad5d3d804e; ghcr.io/hatchet-dev/hatchet/hatchet-dashboard:latest @ sha256:d2ec11aa288a64ce5078054ca4fdcf52bd604c48f65398d4d9a293c2b7c73c15 (resolves to the same digest as the pinned v0.106.5 tag). Docker Compose deployment, SERVER_MSGQUEUE_KIND=postgres, no CPU/memory limits on any container (host had 62GB RAM, plenty of free headroom throughout). Single worker process, slots=50.

Expected behavior

hatchet-api should either not crash under this load, or — at minimum — be automatically restarted if it does, so that the API becomes available again within seconds rather than staying dead indefinitely until someone notices and manually restarts the container.

Code to Reproduce, Logs, or Screenshots

Task + enqueue path — 50,000 individual task triggers, gated at 100 concurrent in-flight requests:

hatchet = Hatchet()

@hatchet.task(name="b01-task", retries=1, backoff_factor=1.0, execution_timeout=timedelta(minutes=5))
async def task(value: Input, ctx: Context) -> dict:
    await asyncio.sleep(0.3)
    return {"ok": True}

gate = asyncio.Semaphore(100)

async def one(v):
    async with gate:
        await task.aio_run_no_wait(Input.model_validate(v), additional_metadata={"campaign": campaign}, desired_worker_labels=[{"key": "phase", "value": "drain", "required": True, "weight": 100}])

await asyncio.gather(*(one(v) for v in items))  # items = 50_000 payloads

Timeline observed: at 06:19:54.098 the first enqueue_ack fires (burst starts). By 06:20:04 — i.e. ~10 seconds into the burst — nginx (in front of hatchet-api in the same container) starts logging connect() failed (111: Connection refused) while connecting to upstream ... upstream: "http://127.0.0.1:8080/...", meaning hatchet-api had already stopped listening. docker top <hatchet-dashboard container> at this point shows only entrypoint.sh and nginx (master + 4 workers) — hatchet-api (normally PID 7) is entirely absent. docker logs for the container in this window contains no output from hatchet-api itself — only nginx's own access/error log lines; no panic trace, no fatal error:, nothing. The dead API stays dead until externally restarted; nginx never stops serving (and never stops returning 502).

What we ruled out for the crash itself: not OOM — docker exec <container> cat /sys/fs/cgroup/memory.events showed oom_kill 0, host dmesg -T (unfiltered) had no OOM-killer entries in the relevant window, and journalctl -u docker had no relevant entries either (expected: hatchet-api is not the container's PID 1, so Docker/containerd never sees its exit at all). The container itself never restarted (RestartCount=0 throughout) — this is specifically the internal process dying, invisible to Docker's own container lifecycle. We were not able to determine the exact proximate cause of the process exit (no core dump or crash trace was ever produced) — we're reporting the observable symptom plus the structural gap below, not a definitive root cause for why the process exits.

The image's own entrypoint.sh starts hatchet-api once, in the background, and never checks on it again outside of container shutdown:

trap 'shutdown' SIGTERM SIGINT

shutdown() {
  echo "Gracefully shutting down hatchet-api..."
  kill -SIGTERM "$HATCHET_API_PID"
  wait "$HATCHET_API_PID"
  echo "Shutting down NGINX..."
  nginx -s quit
  exit 0
}

./hatchet-api "$@" &
HATCHET_API_PID=$!

# ... nginx config templating ...

nginx -g "daemon off;"

There is a trap for graceful shutdown (container stop), but nothing handles hatchet-api exiting on its own for any other reason.

Suggested fix — wrap the launch in a small supervising loop, e.g.:

run_hatchet_api() {
  while true; do
    ./hatchet-api "$@"
    echo "hatchet-api exited with code $?; restarting in 1s" >&2
    sleep 1
  done
}
run_hatchet_api "$@" &
HATCHET_API_PID=$!

(with whatever adjustment keeps $HATCHET_API_PID/kill/wait in the shutdown() trap working against the supervising loop, or switch to a minimal process manager like s6-overlay/tini+dumb-init with a real restart policy.)

Additional context

This was found while load-testing 50,000 individually-triggered tasks against a SERVER_MSGQUEUE_KIND=postgres deployment (the "simplified" self-hosted option, as opposed to the default RabbitMQ backend in the official docker-compose quickstart). Switching that one deployment to RabbitMQ made the crash disappear entirely across repeated identical runs (50,000/50,000 completed, 0 failures, no restarts) — so the trigger for this particular crash appears tied to SERVER_MSGQUEUE_KIND=postgres under this burst pattern. We're not filing that as a bug on its own (it's a documented "simplified deployment" tradeoff), but flagging it since it's what reproduces this issue in the first place, and because we'd guess any sufficiently large spike of internal errors/timeouts (regardless of source) could trigger the same silent, unsupervised death — the missing supervision is the part worth fixing independent of what causes any particular crash.

• I acknowledge that an LLM was used in the creation of this Issue, in accordance with Hatchet's AI_POLICY.md.

• Details: Claude (Anthropic, Claude Code) was used throughout: writing the reproduction harness, running the load test, live process/log/cgroup forensics (docker top, docker logs, dmesg, journalctl, /sys/fs/cgroup/memory.events) to characterize the crash and rule out OOM, reading entrypoint.sh inside the image to confirm the missing supervision, and drafting this report. All reproduction steps and log excerpts above were actually executed/captured during the session, not fabricated by the model.