Engine assigns a second worker identity to the same connection when REGISTER_WORKER is processed after the function burst; reaping the stranded identity unregisters all functions by name
Summary
When a worker process is started by the engine's own iii-exec module, it typically boots before the engine's worker WebSocket listener accepts connections. If the app registers its functions while the SDK is still disconnected, two defects combine to wipe every registration a few seconds or tens of seconds after boot — permanently in configs without watch:
1. SDK (node): disconnected registrations are sent twice. registerFunction() while disconnected records the function in this.functions AND queues the message in messagesToSend. onSocketOpen replays this.functions and then flushes messagesToSend, so every RegisterFunction goes out twice on the same socket (we measured 534 sends for 267 functions), and the queued REGISTER_WORKER invocation from registerWorkerMetadata() is likewise duplicated by the fresh call at the end of onSocketOpen.
- Engine: second worker identity for one connection + by-name cleanup. The engine assigns a provisional worker id when the connection is accepted; the function burst lands under it. When the
REGISTER_WORKERengine-function invocation is processed later (~2s under boot load), the engine creates a second worker identity for the same live connection and re-attributes every function to it, logging per function:Function ownership transferred between two live workers — possible cross-worker overwrite. The stranded first identity is then cleaned up, and that cleanup unregisters the functions by name, removing the successor's registrations and every HTTP route. NoWorker disconnectedevent is involved, so the owner-aware unregister skip from #1796 never fires — that guard only covers dead-connection cleanup.
Reproduction
- engine 0.19.7 or 0.22.1, node SDK 0.19.7 or 0.22.1 (all four combinations reproduce)
- config: an
iii-execworker with NOwatchkey,exec: [node worker.mjs], where the worker registers a few hundred functions at module load - boot the engine; by t+40s the worker's routes 404 while the worker process is alive and its socket connected
Engine log signature: N× ownership transferred, N× [UNREGISTERED], one more Worker registered event than there are worker processes, zero Worker disconnected, zero Skipping unregister.
With watch present, the file-watch respawn happens to recover the routes, which masks the bug in most real configs.
A wire-level SDK trace (appendFileSync hooks on connect/onSocketOpen/sendMessageRaw) confirms: ONE socket reaches OPEN, and zero UnregisterFunction messages are ever sent by the client — the unregistrations are engine-side.
Impact
- Any watch-less
iii-execworker can lose all its functions permanently after a clean boot: the process stays alive and connected, so nothing external notices except route 404s. iii-execdoes not respawn a worker whose process exits, so there is no self-recovery path in that topology.
Suggested directions
- Engine: keep exactly one worker identity per connection (make REGISTER_WORKER update the existing identity in place rather than creating a successor), or make the stranded-identity cleanup owner-checked the same way #1796 made dead-connection cleanup owner-checked.
- SDK: ononSocketOpen, drop queued RegisterFunction/RegisterTrigger/RegisterTriggerType messages that are already covered by thethis.functions/this.triggersreplay, so registrations made while disconnected are sent exactly once.
Workarounds we ship meanwhile
- Wait for the engine to accept a TCP connection before calling
registerWorker(), so the socket is open before the first registration exists (single-shot sends). - Treat
watchas load-bearing in everyiii-execconfig and document it as such.
Source: iii-hq/iii