Cross-account import interest is permanently lost when an account is introduced by config reload in a cluster
Observed behavior
In a clustered deployment, introducing a new account (with a stream export into an
existing JetStream-owning account) via SIGHUP config reload can permanently leave some
servers without the cross-account import interest. Clients that publish through those
servers have their messages silently discarded at the account boundary: no error to the
publisher, no error on the stream, and nothing logged on the losing path.
The condition survives further config reloads and new subscriptions. Only forcing the
stream's subscription to be re-created with something like nats stream cluster step-down or recreating the server instance fixes it.
Because the exporting account is publish-only, its clients cannot obtain a JetStream publish ack, so the data loss is undetectable from either end.
Expected behavior
Having a stream in one account consuming messages on a topic that is exported from another account, should not lose the interest upon a reloaded account setting via a SIGHUB.
Server and client version
nats-server 2.14.0 (nats:2.14.0-alpine)
Host environment
Reproduced on Linux (Docker, kernel 6.6 WSL2)
Steps to reproduce
accounts_after.txt accounts_before.txt docker-compose.yml node.txt run.sh
A test script is attached, but the steps are roughly
- Cluster with account
IMP(JetStream enabled) owning a stream onapp.*.cmd.>. The publishing user initially lives insideIMP. - The stream's leader is on
n0. - A client connects as that user and publishes continuously.
- Config is changed to move that user into a new account
EXPwhichstream-exportsapp.*.cmd.>toIMP, andIMPimports it. - SIGHUP the servers: the stream leader's server (
n0) first, the others ~40s later. - Interest is lost and not messages reaches the stream.
Submission acknowledgment
- I am a human being writing in my own words and not an AI agent. I will not use an AI agent to communicate on my behalf in this issue, either directly or via copy-paste.
Source: nats-io/nats-server