paperclipai run wedges permanently on an undrained TTY stdout — health stays 200 while every run blocks

paperclipai run wedges permanently when stdout is an undrained TTY — server stays "healthy" while every run blocks

TL;DR — When paperclipai run's stdout is a TTY whose master end is never read, the process blocks forever inside a synchronous write(2) on fd 1 once the terminal's buffer fills. The listening socket stays open and /api/health keeps returning 200, so every port-dialing health check passes while every agent run on the host blocks. There is nothing to alert on.

We hit this on a real instance, spent a while diagnosing it, and have a self-contained reproduction. Filing the report rather than a PR because the fix involves a semantics tradeoff that is yours to make — details at the bottom.

Reproduction

Wedges in seconds. The shape is a bare foreground paperclipai run in a terminal nobody is reading — or any pty whose master end is never drained:

import os, pty, subprocess, sys, time

script, port, seconds = sys.argv[1], sys.argv[2], int(sys.argv[3])
master, slave = pty.openpty()
proc = subprocess.Popen(
    ["node", script],
    stdin=slave, stdout=slave, stderr=slave,
    env={**os.environ, "FAKE_PORT": port},
)
os.close(slave)
print(proc.pid, flush=True)
time.sleep(seconds)   # nothing ever reads `master`
proc.kill()

Start the server under that harness, let it emit roughly 64 KB to stdout, then poll /api/health.

Observed: health returns 200 throughout. The process sits at 0% CPU. Every request that would produce log output hangs forever.

Expected: either the server keeps serving and drops or buffers log output, or it fails loudly — but not "reports healthy while doing nothing".

Evidence

sample on the wedged process: 2169 of 2169 samples parked in uv__try_write → write, on fd 1, pointed at /dev/ttysNNN. Not a deadlock in application code — a synchronous write to a full terminal buffer that never drains.

Why this is worse than it looks

The blast radius is the part we would emphasise. The process keeps its listening socket, so:

• port checks pass
• /api/health returns 200
• CPU is 0%, which reads as idle rather than stuck

Every liveness signal a normal monitor looks at says the service is fine. We only found it by sampling a process we already suspected.

Platform asymmetry, measured

• TTY writes are synchronous on all POSIX platforms. So the undrained-terminal case wedges everywhere.
• Pipe writes are async on macOS, synchronous on Linux. So paperclipai run | tee run.log — a very natural thing to type — is this same defect on Linux, and merely fragile on macOS.

That second line is why we think this deserves a fix rather than a docs note: the workaround most people would reach for reintroduces the bug on the more common server platform.

The tradeoff, which is yours and not ours

We deliberately have not written the patch, because the obvious fixes each give something up:

• Async stdout / drop on backpressure trades the wedge for a lost tail on crash. The lines most worth keeping are exactly the ones a crash before the logger is up produces — which is the case a control-plane-stdout.log exists to preserve in the first place.
• Bounded buffer then drop has the same failure with a bigger window.
• Detect a non-draining fd and refuse to start is loud and safe but changes startup behaviour for anyone currently running in a terminal.

We would rather get agreement on the intended semantics than pick one and throw the work away.

Happy to write the patch once you say which shape you want. The reproduction above is yours to use either way.

Environment

• macOS, Node.js via the bundled runtime
• Reproduced against a local paperclipai run and against a minimal fake server under the same harness, so it is not specific to our application code