Support windowed recoverable stream caches to isolate lagging subscriptions

Summary

Introduce a windowed recoverable stream cache which remains one logical IQueueCache per physical partition but can internally retain multiple non-contiguous ranges of partition history. Nearby subscriptions share a cache window; when one subscription falls far behind and pins the oldest record, the cache can split into lagging and hot windows so that up-to-date subscriptions continue receiving newer records without waiting for the lagging subscription.

This is proposed as a follow-up to #10588 and should not be part of that PR.

Motivation

The recoverable partition pipeline in #10588 intentionally uses correctness-preserving backpressure. The durable partition checkpoint and cache purge watermark are the earliest contiguous position which is safe across all active subscriptions.

For example:

Partition: 100(A), 101(B), 102(B), ... 10,000(B)

Subscription A safe through: 99
Subscription B safe through: 9,999
Durable checkpoint:          99

If A has an unresolved matching delivery at position 100, the current contiguous cache cannot purge position 100 or later records. Once the cache reaches capacity, partition reads stop. B can consume records already in the cache, but cannot receive newer records until A advances or stops participating in the watermark.

This behavior avoids data loss, but creates partition-level head-of-line blocking.

The durable provider already retains uncheckpointed partition history. Therefore, in-memory residency does not need to remain contiguous: memory can retain separate working sets around distinct subscriber positions while durable retention remains governed by the single global checkpoint.

Proposed model

Keep the external model unchanged:

one physical partition
one receiver/cache coordinator
one IQueueCache
one durable partition checkpoint

Internally, allow the cache to manage multiple resident windows and subscription views:

IQueueCache
├── shared pooled record storage
├── resident-range/index manager
│   ├── lagging window [100–300]
│   └── hot window [9,800–10,000]
└── subscription views
    ├── subscription A → lagging window
    └── subscription B → hot window

Shared windows

When subscriber working sets overlap or are close enough, keep one combined window and store each physical record once:

A needs [100–250]
B needs [200–350]

resident shared window: [100–350]

Split windows

When one subscription pins the head and the shared window reaches pressure, split the resident ranges:

before:
[100────────────────────────1,000]
 ↑ A                             ↑ B

 after:
[100──250]                 [1,001──1,750]
 ↑ A                              ↑ B

The omitted range remains in durable provider storage. If A later reaches the tail of its resident window, its range is rehydrated from the provider starting after its current position.

Merge windows

When previously separated subscription groups catch up and their required ranges overlap, merge the windows and deduplicate shared physical records.

Memory allocation policy

Use soft, borrowable quotas rather than permanently dividing memory equally per subscriber.

Suggested initial policy:

1. Begin with one shared hot window.
2. Reserve a minimum capacity for each active window.
3. Allow a window to borrow otherwise unused capacity.
4. When pressure is caused by a subscriber pinning the oldest resident record, move that subscriber and nearby subscribers into a lagging window.
5. Keep the remaining subscribers in the hot window.
6. Merge windows once their resident ranges overlap.
7. Cap the total number of windows.
8. If the cap is reached, apply an explicit policy such as shared backpressure, subscriber faulting, or priority-based allocation.

A first implementation could support only two windows—one lagging/catch-up window and one hot window—before generalizing to multiple lag tiers.

Correctness invariants

The implementation must preserve the following:

1. One global durable checkpoint. The checkpoint remains the minimum safe partition position across participating subscriptions.
2. Memory eviction does not imply durable progress. Removing a record from a resident window must never advance the durable checkpoint.
3. No safe-position gaps. A subscription's safe position cannot cross its earliest unresolved matching delivery.
4. Dropped ranges are recoverable. A non-contiguous memory gap is permitted only when the source can reopen or read retained partition history from the required position.
5. Every pending position is resident or reloadable. If it is no longer retained, surface a retention-gap or DataNotAvailableException rather than silently skipping it.
6. Overlapping windows share storage. The same physical partition record should not be retained multiple times when windows overlap.
7. Provider ordering is authoritative. Window boundaries and positioning use provider-defined token/offset comparison.
8. At-least-once recovery remains valid. A crash may cause duplicate delivery, but must not omit uncheckpointed records.

Recovery behavior

Per-subscription durable checkpoints are not required for correctness.

If the global checkpoint is 100 because A is lagging, while B had reached 10,000 before a crash, recovery can conservatively start after 100. B may receive duplicates while rediscovering its progress. Once B reaches the initial cache tail and A continues to pin the head, the cache can split internally and allow B to continue in a hot window.

Optional durable per-subscription positions could later be introduced as resume hints to reconstruct separated windows immediately and reduce duplicate replay/provider I/O. They should not replace the global correctness checkpoint.

Source/read requirements

A sparse cache needs to fill ranges from multiple positions:

lagging reader: continue after 250
hot reader:     continue after 10,000

The initial implementation should target providers which can reopen retained history at an arbitrary provider offset. ADO.NET partition logs are a natural first target because reads are already expressed as ordered records after a message ID.

Kinesis and Event Hubs may require independently positioned reader sessions and provider-specific limits to avoid excessive duplicate reads, iterator usage, or throttling.

Prefer an additive optional capability over changing established queue/cache APIs, for example an independently positioned reader/source capability. Providers which do not support it should retain the existing single-window behavior.

Public API expectations

Windowing should remain transparent to applications and existing queue abstractions:

• IQueueAdapterCache still creates one cache per queue.
• IQueueCache still represents the full physical partition cache.
• IQueueCacheCursor remains a logical subscription view.
• IQueueCache.UpdateDeliveryProgress still receives the global earliest-safe watermark.
• IQueueAdapterReceiver remains one receiver/coordinator per partition.
• IStreamCheckpointStore continues storing one global partition checkpoint.

Do not expose window IDs, split/merge decisions, quotas, or subscription-to-window assignment publicly.

The recoverable source implementation may need an additive optional interface for independently positioned reads. Cursor rehydration state can remain internal.

Suggested implementation phases

Phase 1: ADO.NET two-window prototype

• Add one hot window and at most one lagging window inside the existing recoverable queue cache/coordinator.
• Split only after actual cache pressure and a demonstrably pinned oldest record.
• Rehydrate lagging ranges using ordered ADO.NET reads after an explicit message ID.
• Keep one global checkpoint and existing retention semantics.

Phase 2: generalized window manager

• Group subscribers with nearby positions.
• Support multiple lag tiers with a configured maximum.
• Add soft quotas, borrowing, merge thresholds, and hysteresis to avoid split/merge thrashing.
• Add metrics for window count, resident ranges, rehydration reads, duplicate reads, and per-window pressure.

Phase 3: additional providers

• Define provider capability and concurrency limits for independently positioned readers.
• Evaluate Kinesis shard iterator/read limits and Event Hubs reader costs before enabling multiple windows.

Test scenarios

• A slow matching subscriber pins an old record while a fast subscriber continues beyond total cache capacity.
• A quiet subscription safely scans unrelated records and does not cause a split.
• Two nearby subscriber views share one physical window and one copy of each record.
• A pinned subscriber causes a split; the fast window continues while the global checkpoint remains pinned.
• A lagging window reaches its tail and rehydrates the missing durable range without omission.
• Split windows merge after the lagging subscriber catches up.
• Overlapping reads are deduplicated.
• Failure during window admission does not advance any read or safe position.
• Recovery from only the global checkpoint remains correct and may redeliver to the previously hot subscriber.
• An optional stale or missing per-subscription resume hint only increases duplicate work and cannot skip delivery.
• Hard retention deleting a required nonresident range produces an explicit retention-gap failure.
• Window count and quota limits prevent unbounded memory or reader growth.
• Split/merge hysteresis prevents oscillation under borderline lag.

Acceptance criteria

• A lagging subscription no longer necessarily prevents an up-to-date subscription on the same physical partition from receiving newer records after the original shared window reaches capacity.
• The global durable checkpoint never advances beyond the earliest safe subscription position.
• Total resident memory remains within the configured cache budget.
• Overlapping subscription views share physical cached records.
• Nonresident retained ranges can be rehydrated without skipping records.
• Providers without independently positioned read support retain current contiguous-cache behavior.
• Existing application-facing streaming and queue-cache APIs remain compatible.
• The feature includes diagnostics and metrics sufficient to explain window splits, merges, pressure, and rehydration.