JetStream: ~20s delivery delays through sourced streams after native pool-placement migration

Observed behavior

During 3→6-node upscale and 6→3-node downscale, native placement changes produced 16–20s end-to-end message delays. Manual follower-first movement, including dependent R-leader refresh, reduced maxima to 1.760s up / 2.574s down at the same traffic rates.

Data path: producer → write stream W → sourced read stream R → consumer. Replication stayed configured at R=3, with traffic running throughout. All reported cycles completed, all expected receipts arrived, and final integrity checks passed.

Full-layout results: NATS 2.15.0-RC.1

All rows use 128 W/R pairs, 100,000 subjects and five readers. Rates are aggregate; 2,500 deliveries/s means five receipts per update, not 2,500 clients.

Method / clients	Action	Updates/s → deliveries/s	Total scaling time	Delivery p50	Delivery p99	Maximum message delay
Native / Go	3→6	500 → 2,500	22m53.133s	≤2ms	≤437ms	16.480174s
Native / Go	6→3	100 → 500	9m58.161s	≤2ms	≤12ms	19.255440s
Native / Python	3→6	500 → 2,500	23m30.58s	≤3ms	≤393ms	18.824s
Native / Python	6→3	100 → 500	10m37.15s	≤3ms	≤64ms	19.890s
Manual follower-first + R-leader refresh / Python	3→6	500 → 2,500	16m11.26s	≤5ms	103.97ms	1.760s
Manual follower-first + R-leader refresh / Python	6→3	100 → 500	15m37.45s	≤3ms	80.16ms	2.574s

Each method/client combination is one completed cycle. All finished with healthy streams and no observed missing receipts, corruption or older-generation regression.

Latency is scheduled publication → actual consumer callback, including dispatch and retries. Go first-send-to-receipt maxima were nearly identical: 16.479377s / 19.255313s. Maxima are actual message delays, not verification waits. Percentiles marked ≤ use 1ms-ceiling buckets. Duration includes checks and balancing, excluding setup, baselines, final receipt settling and cleanup.

Go baseline maxima before up/down were 242.900ms / 15.467ms. During scaling, 6,733 / 465 deliveries exceeded 10s. All 3,432,835 / 299,085 receipts arrived. Retries occurred overall, but neither maximum-delay message needed a retry.

Expected behavior

Placement changes between healthy pools should preserve prompt sourced-stream delivery without external leader manipulation.

Server and client version

• Full-layout runs: nats-server 2.15.0-RC.1, verified from the running binary. All nodes used synadia/nats-server@sha256:0d18bb3aa53dd6aea04170b2696c8246331809114b23e29f4f1e3cb2b927866b.
• Go traffic run: Go 1.26.3, nats.go 1.53.1. Python comparison runs/controller: nats-py 2.11.0. NATS CLI 0.4.0 for balancing.
• Also observed on 2.14.5 in smaller completed cycles below.

Host environment

• Shared Linux Docker host: Intel i9-13900H, 20 logical CPUs, ~62.5GiB RAM, no swap, local disk. AZs are tags on one host, not separate hardware.
• Cluster region1, account APP. Pool 1: r1-1/2/3; pool 2: r1-4/5/6. Each pool has one node per az:a/b/c, plus pool:N; unique_tag: "az".
• 128 W/R stream pairs, file storage, R=3, max_msgs_per_subject: 1. W/R pairs share their final pool.
• W captures odds-v4.<partition>.r1.>; R sources its corresponding W. No cross-region sources.
• Two producer connections; five reader connections, 640 R3 ephemeral push consumers, AckNone / DeliverNew / ReplayInstant, partition filters, one-hour inactivity threshold.
• 100,000 subjects. Payload min/p50/mean/p99/max: 1,684 / 1,788 / 3,312 / 25,296 / 29,752 bytes.
• Per-subject serialized publishing; 1s timeout, 100ms retry pause, 30s budget. Retries preserve payload, message ID and original timestamp; W deduplication window 60s.

Steps to reproduce

Upscale, with 500 updates/s running continuously

1. Start pool 1, create/seed the streams and attach readers. Verify data; measure a 30-second traffic baseline.
2. Add pool 2's three nodes sequentially. Wait for health and current metadata membership.
3. Compute deterministic placement for two pools: 68 W/R pairs move, 136 streams. Process ascending partition number, W completely before its R, one stream at a time.
4. Send one $JS.API.STREAM.UPDATE.<stream> with the existing configuration, changing only placement from {"cluster":"region1","tags":["pool:1"]} to {"cluster":"region1","tags":["pool:2"]}. This selects all three destination nodes, not a chosen leader; num_replicas stays 3.
5. Let NATS manage migration. Samples showed up to six old/new peers temporarily. Wait for exactly three current destination peers, zero replica lag, healthy consumers and cleared cluster.desired; verify W→R payload delivery and global health/storage before the next stream. Polling: 10→20→40→80→100ms, plus request time.
6. After all moves, run nats stream cluster balance, verify final placement/health, and check every expected delivery and latest value.

No manual peer replacements, unpinning, intermediate leader handoffs or dependent-R leader refreshes were performed in native mode.

Downscale, with 100 updates/s running continuously

1. Measure a 30-second lower-rate baseline; verify that pool 2 contains only accounted-for streams.
2. Move the same 136 streams, in the same W-then-R order, by changing only pool:2 back to pool:1. Apply the same completion and delivery checks after each stream.
3. Prove pool 2 has no remaining stream/consumer replicas or storage to preserve. Only then retire its nodes sequentially: stop, metadata peer-remove, and remove owned containers/storage, checking surviving health between nodes.
4. Balance leaders and verify all W/R streams, final data and all expected receipts.

What the worst-message traces show

Go run, 11 September. Times are relative to the affected message's scheduled publication (t=0); negative times precede it.

Event	Upscale: W_15 → pool 2	Downscale: W_108 → pool 1
W placement UPDATE requested	−0.753s	−0.833s
New W leader first observed	−0.280s	−0.358s
Successful PubAck, first attempt	+1.289ms	+0.509ms
W native migration considered settled	+0.252s	+0.048s
Last R message-get probe still returned an older value	+14.234s	+18.869s
Slowest actual reader receipt	+16.480174s	+19.255440s
Corresponding R placement UPDATE requested	+17.480s	+19.902s

Both messages were promptly acknowledged but remained unavailable through R long after W migration settled. R had not yet moved; node removal and final balancing had not occurred. This supports investigating W→R recovery, but does not identify the internal cause or exact ingestion time.

Manual follower-first comparison: what changed

The same pool lifecycle and W-then-R ordering were used, but each stream moved as follows:

1. Record the current leader and followers; clear the pool restriction with placement: null, keeping R=3 and confirming the existing peers remain.
2. Replace each non-leader replica first, individually, with the destination node in the same AZ. Use the system-account API $JS.API.ACCOUNT.STREAM.MOVE.APP.<stream>, e.g. {"server":"r1-1","cluster":"region1","tags":["pool:2","az:a"]}. server is the old peer to replace, not the destination. Each replacement transitions 3→4→3 peers; wait for exact membership and caught-up stream/consumer replicas before continuing.
3. Before removing a peer hosting replicated consumer leadership, hand that leadership to a retained current stream peer. Before replacing the old stream leader, transfer stream leadership to a caught-up destination peer, then replace the old leader's copy.
4. Wait for cluster.desired reconciliation to finish; restore the destination pool pin.
5. After moving W, transfer its dependent R stream's leadership to a current follower and verify W→R delivery. Then move R. Reverse the destination pool for downscale; remove nodes only after full evacuation.

This combines follower-first replacement and R-leader refresh; we have not isolated which change accounts for the improvement.

Supporting evidence and limitations

• 2.14.5 also reproduced the symptom: 8 partitions, 1,000 subjects, ten Go producers and five readers using default-replication ephemeral consumers; 50 updates/s up, 10/s down. Native maxima 16.544s / 18.437s; follower-first 1.799s / 1.486s. Both cycles completed, all receipts arrived, zero retries.
• Single runs on shared hardware, not controlled A/B benchmarks or Kubernetes/capacity qualification. Clients, polling and verification differ; follower-first used fixed 100ms polling.

Submission acknowledgment

• I am a human being writing in my own words and not an AI agent. I will not use an AI agent to communicate on my behalf in this issue, either directly or via copy-paste.