Federation: drain a link's buffer responsively and drop messages before buffering them
[!NOTE] This issue was drafted by Claude (Anthropic's Claude Code) under the direction of @lukebakken. The code references were verified against the branch of #17238. The work described already exists on a local branch; what it lacks is test coverage, which is why it was removed from that pull request.
#17238 buffers deliveries while the downstream node has a resource alarm. Two changes were written as part of it and then removed, because neither is needed to close #16670 and neither is observed by a test.
Drain responsiveness
rabbit_federation_link_util:drain_buffer/4 forwards the whole buffer inside one handle_info callback. A link draining up to prefetch-count messages answers nothing in between: not a basic.ack, not a confirm, not its downstream channel dying, not a supervisor shutdown. With ack-mode on-confirm it also makes one synchronous amqp_channel:next_publish_seqno/1 call per buffered message without returning to its mailbox.
The approach was to forward one delivery per callback and re-arm via a continue_drain message. Neither link module implements prioritise_info/3, so gen_server2 gives every message priority 0 and processes the re-arm behind anything already queued, which is what lets the link answer those messages mid-drain. It requires a delivery arriving mid-drain to be buffered too, or it would be forwarded past messages still queued.
Pre-buffer drop filter
max-hops and the loop check are evaluated in forward/9, which for a buffered message runs only at drain time. A message the link was always going to drop therefore occupies buffer space and holds a slot in the upstream's prefetch window until then.
Deciding before buffering also means resolving the cluster name twice per delivery, so it should be resolved once and threaded through both checks. rabbit_nodes:cluster_name/0 reads ETS only when the cluster_name runtime parameter is set, and otherwise resolves the local hostname.
Why it was removed
Nothing observes the responsiveness. Restoring the whole-buffer loop leaves every test in exchange_alarm_SUITE and queue_alarm_SUITE green, so the property rests on gen_server2's message ordering rather than on an assertion.
The queue link's share of the ordering guard is also uncovered: queue federation pulls from upstream only while the downstream queue is empty with an active consumer, so a second batch is not consumed mid-drain there.
What to do
Restore both changes and add the coverage they lack:
- An assertion that a link answers something, an ack, a confirm, or a status change, while a long buffer is still draining. This is the hard part and the reason the work was deferred.
- Coverage of the queue link's ordering guard, which needs a way to get queue federation to consume a second batch mid-drain.
Ordering on the exchange side is already covered by deliveries_during_a_drain_keep_their_place: 200 messages buffered under an alarm, 50 more published into the drain window, all 250 asserted in publish order.
Source: rabbitmq/rabbitmq-server