Stream sourcing throughput capped at ~20K msg/s by a single goroutine

Proposed change

Summary

JetStream's stream-sourcing feature (Sources config on a stream, used directly and via Orbit's pcgroups elastic Partitioned Consumer Group library) funnels all inbound sourced messages for a destination stream through a single goroutine (processAllSourceMsgs in server/stream.go). Across multiple independent load tests on a 3-node R=3 cluster, we consistently measured a hard sourcing throughput ceiling of ~18,300–21,500 msg/s, regardless of:

• Doubling the underlying hardware (8 vCPU → 16 vCPU per node)
• Doubling the number of downstream consumers (50 → 100 elastic PCG members)
• Increasing the sourced (destination) stream's buffer size (2 GiB → 26 GiB)
• Increasing the source stream's retention window (max-age 30m → 2h, max-bytes 20 GiB → 35 GiB)

Meanwhile the cluster nodes themselves were confirmed to have substantial CPU headroom (20-50% utilization observed during these tests, well below saturation), ruling out a hardware/resource constraint. The bottleneck is architectural: the sourcing engine's own code acknowledges this with an explicit TODO.

This single-goroutine ceiling, combined with discard=old retention semantics on the source stream, causes silent, permanent, unrecoverable message loss at any publish rate that exceeds ~18-21K msg/s sustained — with no error, warning, or metric surfaced anywhere in the API to indicate that messages were dropped before ever being read by the sourcing consumer.

My bigger concern is my end-to-end throughput is now restrained at the throughput I can get out of the sourcing single goroutine, which per my performance tests during POC was about 20K msgs/sec. My publish throughput is 70K msgs/sec, which is the expected throughput on production for my product, however, due to slow sourcing when using Orbit library, my subscriber throughput is way less than the expected 70K.

Please let me know if this is a known issue and if there is any plan on improving the concurrency or performance of sourcing mechanism. This is a major blocker for our use case, and could lead us to start looking for some other solution. We are currently in POC phase for our product.

Details can be found below.

Environment

• NATS Server: v2.12.6
• nats CLI: v0.3.2
• Cluster: 3-node R=3 cluster (bench-cluster), tested on both m8g.2xlarge (8 vCPU/32GiB) and c8g.4xlarge (16 vCPU) Graviton4 instances
• Client library used for sourcing consumption: github.com/synadia-io/orbit.go/pcgroups (Elastic Partitioned Consumer Group), v0.2.2 — though the bottleneck is entirely inside nats-server itself; pcgroups.CreateElastic() is a thin wrapper that just calls js.CreateStream() with a native Sources block. No Orbit-specific logic is involved in the sourcing path.

Probable Root Cause (confirmed from nats-server source)

server/stream.go, lines 4671-4675:

// This will process all inbound source msgs.
// We mux them into one go routine to avoid lock contention and high cpu and thread thrashing.
// TODO(dlc) make this more then one and pin sources to one of a group.
func (mset *stream) processAllSourceMsgs() {

Every message sourced into a destination stream — regardless of how many upstream sources are configured, how many partitions/subject-transforms are in play, or how many downstream consumers exist — is processed through this single goroutine per destination stream.

Test Details

Topology: Single source stream BENCH (R=3, retention=limits, discard=old), sourced into a destination work-queue stream BENCH-cg-bench (R=3) via Orbit's elastic PCG (max-members=100, 100 active members), using a partitioned subject transform (bench.data.* → {{Partition(100,1)}}.bench.data.{{Wildcard(1)}}).

Load: 250,000 concurrent publisher connections publishing at a sustained ~77,000-78,000 msg/s target rate, 3-hour soak, 100-byte payloads.

Observed Data

We measured the destination stream's own Last Sequence growth (the only reliable ground-truth for actual sourcing throughput) across four independent test runs, at two different hardware tiers, with member counts ranging from 50-100:

Happy to share full test logs, nats stream info / nats consumer info snapshots, pprof CPU profiles, and the orchestration scripts used to reproduce this at scale, if useful.

Use case

Any use case relying on JetStream's native stream-sourcing feature — including Orbit's Elastic Partitioned Consumer Groups, which depend entirely on this mechanism — is limited to roughly 18,000-21,000 msg/s of sourced-in throughput per destination stream, regardless of cluster size, hardware, or the number of downstream consumers. At higher publish rates against a source stream using bounded retention (max-age/max-bytes, a completely standard/recommended configuration), this results in silent, permanent, and unmonitorable data loss.

Contribution

No response

Submission acknowledgment

• I am a human being writing in my own words and not an AI agent. I will not use an AI agent to communicate on my behalf in this issue, either directly or via copy-paste.