Federation link status does not show when a link cannot forward

[!NOTE] This issue was drafted by Claude (Anthropic's Claude Code) under the direction of @lukebakken. The code references were verified against the branch of #17238, and the design below is the one rabbit_shovel_status already implements rather than a novel proposal.

A federation link reports running while it is buffering behind a resource alarm on the downstream node, and while credit flow is throttling it. The only external sign is that messages stop arriving: the buffered messages are in no queue's messages count, in no status field, and gen_server2:drain/1 empties the mailbox eagerly so process_info(Pid, message_queue_len) reads about zero.

Every comparable path already reports this. rabbit_shovel_status defines blocked_status() :: running | flow | blocked and surfaces it through rabbitmqctl shovel_status and the management UI. Federation reports {running, ConnName} once, in rabbit_federation_link_util:start_conn_ch/5, and never revises it.

This is the observability half of the work discussed on #17238, which fixes the underlying alarm-blocking bug from #16670. It is filed separately because it did not converge as part of that pull request.

What to build

The shovel already contains the whole design. Read these three files end to end before writing anything, rather than consulting them for the concept: deps/rabbitmq_shovel/src/rabbit_shovel_status.erl (the #entry{}, report_blocked_status/2, and especially blocked_status_to_info/1), deps/rabbitmq_shovel/src/rabbit_shovel_worker.erl (maybe_report_blocked_status/1 and its call sites), and deps/rabbitmq_shovel_management/src/rabbit_shovel_mgmt_util.erl lines 72-73 (the presentation fold).

1. Three values, running | flow | blocked. Two is not enough: blocked means a resource alarm and warrants red, flow means credit-flow throttling and warrants yellow. Collapsing them makes routine downstream slowness indistinguishable from an alarm, and fmt_object_state in the management UI already colours both correctly.

2. Store blocked_status and blocked_at, and apply the decay when the entry is read, using credit_flow:state_delayed/1. That function is exported for exactly this reader. Resolving the hold at write time freezes it permanently, because credit_blocked_at is only ever set and never erased, not even by credit_flow:unblock/1 (see deps/rabbit_common/src/credit_flow.erl).

3. Report the raw credit_flow:blocked/0 from the link. The one-second hold belongs to the reader, not the writer.

4. Report from every path that can change the value, including both branches of the delivery clause. The shovel reports after every message. A missed report then decays instead of latching, which is what makes the design robust against a link having many exit paths.

5. Leave status alone. rabbit_federation_prometheus_collector:collect_mf/2 emits it as a Prometheus label value, so folding flow or blocked into it would silently zero any existing alert on rabbitmq_federation_links{status="running"} during routine throttling. Eight existing test helpers also compare status =:= running, including two bare ?assertEqual(6, count_running_links(...)) calls in triangular_bidirectional_exchange_federation_SUITE.

6. Fold into a single state key at the presentation layer only, as rabbit_shovel_mgmt_util does. Do not derive it inside rabbit_federation_status:format/1 for non-running entries: #entry.status can hold {{badmatch, _}, Stacktrace} via the catch-all clause of log_terminate/4, so any function matching on it must be total or the status server dies and takes its private ETS table with it.

Two traps

Do not clear #state.blocked anywhere in the link modules. It is what is_blocked/1 feeds to rabbit_federation_link_util:drain_buffer/4, so clearing it re-arms the drain. In handle_cast(disconnect_for_shutdown, ...) that drains onto a channel whose connection is closing, which exits on the amqp_channel:next_publish_seqno/1 call in on-confirm mode and silently loses the message in the other ack modes, because cast_flow to a dead pid is a no-op and forward/9 then acks upstream.

Any function that matches on #entry.status must be total, for the reason in point 6 above.

Testing

flow cannot be produced by exchange_alarm_SUITE or queue_alarm_SUITE: they publish 50 messages against a default initial credit of 400, so credit flow never blocks. It can be forced directly in deps/rabbitmq_federation_common/test/unit_SUITE.erl, which can start rabbit_federation_status itself and use credit_flow:block/1. The read-time decay in point 2 also needs a unit test, because forcing credit flow makes credit_flow:state/0 answer from its first clause and never reach state_delayed/1.

Both status-command suites assert the CLI output map with Erlang map patterns, which match supersets, so adding a key there is silent unless it is asserted explicitly. federation_mgmt_SUITE exercises the whole HTTP path but reads only exchange, upstream and status.

Affected modules

deps/rabbitmq_federation_common/src/rabbit_federation_status.erl, deps/rabbitmq_federation_common/src/rabbit_federation_link_util.erl, deps/rabbitmq_exchange_federation/src/rabbit_federation_exchange_link.erl, deps/rabbitmq_queue_federation/src/rabbit_federation_queue_link.erl, deps/rabbitmq_federation_management/src/rabbit_federation_mgmt.erl, and deps/rabbitmq_federation_management/priv/www/js/tmpl/federation.ejs.