#2050·iii

Engine assigns a second worker identity to the same connection when REGISTER_WORKER is processed after the function burst; reaping the stranded identity unregisters all functions by name

Author: robbierobsCreated Aug 8, 2026Updated Aug 11, 2026

Summary

When a worker process is started by the engine's own iii-exec module, it typically boots before the engine's worker WebSocket listener accepts connections. If the app registers its functions while the SDK is still disconnected, two defects combine to wipe every registration a few seconds or tens of seconds after boot — permanently in configs without watch:

1. SDK (node): disconnected registrations are sent twice. registerFunction() while disconnected records the function in this.functions AND queues the message in messagesToSend. onSocketOpen replays this.functions and then flushes messagesToSend, so every RegisterFunction goes out twice on the same socket (we measured 534 sends for 267 functions), and the queued REGISTER_WORKER invocation from registerWorkerMetadata() is likewise duplicated by the fresh call at the end of onSocketOpen.

  1. Engine: second worker identity for one connection + by-name cleanup. The engine assigns a provisional worker id when the connection is accepted; the function burst lands under it. When the REGISTER_WORKER engine-function invocation is processed later (~2s under boot load), the engine creates a second worker identity for the same live connection and re-attributes every function to it, logging per function: Function ownership transferred between two live workers — possible cross-worker overwrite. The stranded first identity is then cleaned up, and that cleanup unregisters the functions by name, removing the successor's registrations and every HTTP route. No Worker disconnected event is involved, so the owner-aware unregister skip from #1796 never fires — that guard only covers dead-connection cleanup.

Reproduction

  • engine 0.19.7 or 0.22.1, node SDK 0.19.7 or 0.22.1 (all four combinations reproduce)
  • config: an iii-exec worker with NO watch key, exec: [node worker.mjs], where the worker registers a few hundred functions at module load
  • boot the engine; by t+40s the worker's routes 404 while the worker process is alive and its socket connected

Engine log signature: N× ownership transferred, N× [UNREGISTERED], one more Worker registered event than there are worker processes, zero Worker disconnected, zero Skipping unregister.

With watch present, the file-watch respawn happens to recover the routes, which masks the bug in most real configs.

A wire-level SDK trace (appendFileSync hooks on connect/onSocketOpen/sendMessageRaw) confirms: ONE socket reaches OPEN, and zero UnregisterFunction messages are ever sent by the client — the unregistrations are engine-side.

Impact

  • Any watch-less iii-exec worker can lose all its functions permanently after a clean boot: the process stays alive and connected, so nothing external notices except route 404s.
  • iii-exec does not respawn a worker whose process exits, so there is no self-recovery path in that topology.

Suggested directions

  • Engine: keep exactly one worker identity per connection (make REGISTER_WORKER update the existing identity in place rather than creating a successor), or make the stranded-identity cleanup owner-checked the same way #1796 made dead-connection cleanup owner-checked. - SDK: on onSocketOpen, drop queued RegisterFunction/RegisterTrigger/RegisterTriggerType messages that are already covered by the this.functions/this.triggers replay, so registrations made while disconnected are sent exactly once.

Workarounds we ship meanwhile

  • Wait for the engine to accept a TCP connection before calling registerWorker(), so the socket is open before the first registration exists (single-shot sends).
  • Treat watch as load-bearing in every iii-exec config and document it as such.