Clustered stream restore failure leaves an orphaned Raft node, and a later re-restore of the same stream can silently disappear [v2.12.15]

Observed behavior

We migrated about 15 R3 JetStream deployments with per-stream nats stream restore. When a clustered restore failed and we then deleted the half-created stream, the cluster was left in a bad state. We saw two different outcomes:

A. Orphaned Raft node that never stops. If the delete (nats stream rm -f) runs right after a failed restore, one member logs this:

[WRN] JetStream failed to lookup running stream while removing stream 'APP > tap' from this server

About 0.1s later, on the same server, a Raft node for the removed group starts logging errors every few seconds. Its directory is gone, and the node keeps running until the server restarts:

[ERR] RAFT [EwpEUKJe - S-R3F-P4IaMiIo] Resource not found: rename /data/jetstream/SYS/_js_/S-R3F-P4IaMiIo/tav.idx.tmp /data/jetstream/SYS/_js_/S-R3F-P4IaMiIo/tav.idx: no such file or directory
[WRN] RAFT [EwpEUKJe - S-R3F-P4IaMiIo] Error writing term and vote file for "S-R3F-P4IaMiIo": rename ...: no such file or directory
[WRN] RAFT [EwpEUKJe - S-R3F-P4IaMiIo] Not requesting votes for "S-R3F-P4IaMiIo", could not persist term and vote: rename ...: no such file or directory

• The group appears in no stream's /jsz output on any server.
• On one deployment it logged about 180 lines per 5 minutes for 21 hours.
• On another, two failed attempts on one stream left two such groups on the same server, at about 1.6k ERR/WRN lines per hour.
• A retry of the stream succeeded, and the live stream runs on a new group.
• Restarting that one server clears the node.

B. A successfully re-restored stream disappears later, with nothing logged. This happened on 3 deployments, each on a different stream:

1. The first restore of stream X failed with is stalled (details below).

2. nats stream rm -f X was sent. The server logged failed to lookup running stream while removing stream 'APP > X', but the client never got a reply (it waited until interrupted; --timeout was 10m).

3. A new run restored X again. It reported success, the replicas caught up, and the message count matched the snapshot exactly (248,455 / 247,870 / 1,090,876 messages).

4. Later, X no longer existed on any server:

   • it was missing from /jsz on all three servers and from nats account backup;
   • clients got stream not found.

   No server logged anything about removing it: no Resetting stream cluster state, no rejected by assigned member, no restore or delete warnings. We know it was gone within about 22 hours (the earliest client logs we still had), and on one deployment within 6 days.

On one deployment, we restarted the server that held the orphaned node from (A). A stream X then reappeared, but empty, with a new Raft group. It was created during that restart, and we don't know what created it.

On the other deployments no streams are missing. That includes deployments where a restore failed with the error from (A) and the retry in the same run, after a stream rm that did reply, succeeded.

Trigger: the restore stall

The first failure in (B) was always the 5s activity watchdog in processStreamRestore:

[WRN] Internal subscription on "$JS.SNAPSHOT.RESTORE.<stream>.8AsmW6ts8ndukuazgSgzjR" took too long: 5.000837224s
[WRN] <route or client> - Readloop processing time: 5.000858364s
[WRN] Stream restore failed for 'APP > X': restore for stream 'APP > X' is stalled

The client saw:

nats: error: restore failed: restore failed: "{\"type\":\"io.nats.jetstream.api.v1.stream_create_response\",\"error\":{\"code\":500,\"err_code\":10062,\"description\":\"restore failed: io: read/write on closed pipe\"}}"

• Each time, the restore of X (about 130 MB) started about 100ms after the previous stream (110–130 MB, over 2M messages) finished restoring.
• It landed on a follower that was still catching up that previous stream (the catchup took 5–11s), on a gp3 EBS volume.
• The pw.Write in processChunk then blocked for more than 5s.
• Waiting until every replica of the previous stream is current before starting the next restore avoided the overlap in our tests.

The restore failures that came before (A) were a different client error:

nats: error: restore failed: server response is not a valid "io.nats.jetstream.api.v1.stream_restore_response" message: /type: value must be "io.nats.jetstream.api.v1.stream_restore_response"
missing properties: 'error'
/deliver_subject: length must be >= 1, but got 0

Expected behavior

• Deleting a stream whose clustered restore failed stops and removes that group's Raft node on every member. It doesn't leave a node running with a deleted directory.
• stream delete always replies.
• A stream that was restored successfully, and caught up, is never removed by leftover state from an earlier failed restore of the same name.

What we checked in the code

We compared v2.12.15 with v2.15.0 and main:

• processClusterDeleteStream is unchanged. When acc.lookupStream returns nil on a member, it only warns. It then deletes sa.Group.node. We suspect this is not always the node that is still running, but we haven't confirmed it.
• The stall branch of processStreamRestore returns without calling finish(), so the restore request gets no reply. The goroutine running RestoreStream is not waited on.
• On a failed clustered restore, monitorStream calls n.Delete() and sends a streamAssignmentResult. That result identifies the stream only by account and name, and processStreamAssignmentResults looks up the assignment by name. That could hit a newer assignment with the same name. We never saw its rejected by assigned member warning, though, so we're not claiming this is the path.
• #8501 (catchup flow control broken since #6944) is not in 2.12.x, and it may explain why follower catchup was heavy enough to stall the restore.
• #8643 (missing API responses during meta catchup) may be related to the delete that never replied.

Server and client version

• nats-server v2.12.15 (8460a42, go1.25.12), nats:2.12.15-alpine
• NATS CLI v0.2.4 (natsio/nats-box:0.18.1)

Host environment

• Kubernetes (EKS on AWS Outposts), one 3-node cluster per deployment, official Helm chart.
• File storage on gp3 EBS (32Gi per server, baseline 3000 IOPS / 125 MB/s).
• CPU limit 4 (gomaxprocs 4), GOMEMLIMIT 6GiB.
• About 80 streams per account, mostly R3, a few R1. disk_io_wait_stats are all zero.

Steps to reproduce

We don't have a deterministic reproduction. The sequence that produced it:

1. Take nats account backup of an account with a few large streams (over 100 MB and over 1M messages each).
2. On a fresh R3 cluster, run nats stream restore <dir> for each stream directory, back to back. The stall needs a follower that is still catching up the previous stream. On an idle cluster we saw the overlap without a stall.
3. When a restore fails, run nats stream rm -f <stream>, then restore the stream again (in the same run, or after the rm hangs and is interrupted).
4. Watch for:
   • a RAFT tav.idx ENOENT loop on one member;
   • the re-restored stream going missing later.

We have full server logs from all three servers for each affected deployment. Timestamps line up across the servers and the client output.