Failover not triggered when primary VM dies mid-`configure` (or any non-`wait` label)

Summary

When the primary Postgres VM becomes unreachable while its strand is executing any label other than wait (e.g. configure, setup_hugepages, configure_metrics), the unplanned-failover machinery never activates. The strand crash-loops on SSH timeouts at the failed label and the standby continues to sleep in wait indefinitely. The resource keeps reporting display_state=running even though the actual primary is gone.

Why it happens

PostgresServerNexus#unavailable is the only place trigger_failover(mode: "unplanned") is called (prog/postgres/postgres_server_nexus.rb:715). The only path into unavailable is from the wait label via the checkup semaphore (postgres_server_nexus.rb:628-635):

label def wait
  ...
  when_checkup_set? do
    unless available?
      register_deadline("wait", 5 * 60)
      hop_unavailable
    end
    decr_checkup
  end
  ...
end

The monitor correctly sets incr_checkup on the primary's strand when SSH starts failing — but this semaphore is read only at the top of wait. Labels like configure start with vm.sshable.d_check(...) (line 452); when SSH times out, the strand throws Net::SSH::ConnectionTimeout, retries the same label, and never returns to wait.

Net result:

1. hop_unavailable is never reached.
2. trigger_failover is never called.
3. The standby never receives unplanned_take_over.
4. The resource is permanently broken from the user's perspective; only manual intervention on the rails console recovers it.

Reproduction

The cleanest repro uses the failure-injection endpoint from our internal fork (https://github.com/ClickHouse/ubicloud/pull/251), but the bug is in upstream code — you can reproduce it on stock ubicloud by killing the primary EC2 instance via the AWS console at the right moment, or by ssh ... sudo shutdown -h now to the primary.

# 1. Provision an HA-enabled Postgres resource and wait for `running`.
NAME=stuck-failover-repro
curl -X POST .../project/$PROJ/location/$LOC/postgres/$NAME \
  -d '{"size":"m8gd.large","storage_size":118,"ha_type":"async"}'
# wait until display_state == running

# 2. Trigger any operation that pushes the primary's strand back into a
#    non-`wait` label. A PATCH that bumps a config or HA type works
#    (each `update` audit row implies an `incr_configure` on the primary):
curl -X PATCH .../project/$PROJ/location/$LOC/postgres/$NAME \
  -d '{"ha_type":"sync"}'

# 3. While the primary is in `configure` (10-30s window), make its VM
#    unreachable. With the failure-injection endpoint:
curl -X POST .../project/$PROJ/clickgres-testing/$NAME/inject-failure \
  -d '{"failure_type":"os_shutdown"}'
# Without it, e.g.:
#   ssh ubi@<primary-ip> sudo shutdown -h now
# or terminate the EC2 instance via the AWS console.

Observed (Clickhouse test environment)

• audit_log for the resource: create, recycle, 3× update, then inject_failure_os_shutdown (at 02:41:39).
• Ten minutes later:
  • Primary strand: label=configure, try=56 — crash-looping on Net::SSH::ConnectionTimeout at postgres_server_nexus.rb:452.
  • Standby strand: label=wait, schedule 6h out, no unplanned_take_over semaphore set.
  • Resource display_state: still running.
  • Monitor: logs a monitoring_job_failure (Net::SSH::ConnectionTimeout) for the primary every cycle. The checkup semaphore it sets is sitting on the primary strand unread.
• TCP connect to primary's :22 times out from anywhere.

Expected behaviour

If the primary VM is persistently unreachable, failover should be triggered regardless of which label the primary's strand happened to be in when SSH started failing.