jetstream sequence inconsistent with actual message order
Observed behavior
Hi, We are running Nats 2.11.3 with Jetstream enabled, on 3 replicas. We are observing (and have no clue how to reproduce) a strange situation in one of our JetStreams : during a few days, messages published and stored in the JS were given lower sequence number than the previous ones.
Running nats sub --stream=MYSTREAM "MYSUBJECT.*" --since=1Y
we get :
[#2053] Received JetStream message: stream: MYJETSTREAM seq: 19,126 / pending: 15,341 / subject: MYSUBJECT.3476709 / time: 2026-06-29 16:27:49 [#2054] Received JetStream message: stream: MYJETSTREAM seq: 19,128 / pending: 15,340 / subject: MYSUBJECT.3476716 / time: 2026-06-29 16:27:55 [#2055] Received JetStream message: stream: MYJETSTREAM seq: 19,131 / pending: 15,339 / subject: MYSUBJECT.3476709 / time: 2026-06-29 16:27:59 [#2056] Received JetStream message: stream: MYJETSTREAM seq: 19,133 / pending: 15,338 / subject: MYSUBJECT.3476716 / time: 2026-06-29 16:28:02 [#2057] Received JetStream message: stream: MYJETSTREAM seq: 19,155 / pending: 15,337 / subject: MYSUBJECT.3488168 / time: 2026-06-29 16:32:44 [#2058] Received JetStream message: stream: MYJETSTREAM seq: 24,439 / pending: 15,336 / subject: MYSUBJECT.3438624 / time: 2026-03-23 19:20:55 [#2059] Received JetStream message: stream: MYJETSTREAM seq: 24,442 / pending: 15,335 / subject: MYSUBJECT.3438626 / time: 2026-03-23 19:21:52 [#2060] Received JetStream message: stream: MYJETSTREAM seq: 24,448 / pending: 15,334 / subject: MYSUBJECT.3438630 / time: 2026-03-23 19:24:43 [#2061] Received JetStream message: stream: MYJETSTREAM seq: 24,453 / pending: 15,333 / subject: MYSUBJECT.3438632 / time: 2026-03-23 19:29:46
message seq 19,155 is actually newer than msg seq 24439 as per timestamp
and many messsages later we can see the real successor of 19155 with a more expected sequence :
[#15572] Received JetStream message: stream: MYJETSTREAM seq: 207,387 / pending: 1,822 / subject: MYSUBJECT.3488199 / time: 2026-06-29 16:59:41 [#15573] Received JetStream message: stream: MYJETSTREAM seq: 207,403 / pending: 1,821 / subject: MYSUBJECT.3488203 / time: 2026-06-29 17:04:37 [#15574] Received JetStream message: stream: MYJETSTREAM seq: 207,409 / pending: 1,820 / subject: MYSUBJECT.3488177 / time: 2026-06-29 17:05:47
We observe also that consumers were able to deliver the messages with seq 19xxx.
-> it is normal behavior? in what circumstances? -> if not how to avoid it and possibly repair the sequence?
thanks
Expected behavior
My understanding was that the Jetstream sequence number was strictly increasing.
Server and client version
Server : 2.11.3 Nats CLI : 0.4 jnats : 2.25.2
Host environment
No response
Steps to reproduce
No response
Source: nats-io/nats-server