[Streaming][Event Hubs]Partition rebalancing can cause replay of messages

In Orleans v10.2.2 using the EventHubStreamProvider, we have observed that it is possible for the partition rebalancing triggered when new silos enter a cluster to result in message replay.

When a pulling agent gives up responsibility for a partition, it flushes its latest checkpoint.

However, we observed that the newly responsible pulling agent began pulling before the previous agent had flushed its checkpoint. This caused the new agent to pull from an older checkpoint, with a large volume of messages being reprocessed as a result.

This was observed using the ConsistentRingQueueBalancer, but we were able to reproduce this issue with a custom balancer triggering the rebalance on demand.

A collaborative handoff when a new pulling agent takes ownership of a partition could potentially address this issue.