Metrics export couples the SSH scrape and the VictoriaMetrics push on one reused session, so a slow backend cascades into fleetwide `IOError: closed stream` scrape failures

Summary

MetricsTargetMethods#export_metrics runs the SSH scrape and the time-series push on the same worker thread, over the same long-lived SSH session. When the metrics backend (VictoriaMetrics) is slow or unavailable, this couples a backend slowdown into fleetwide SSH IOError: closed stream scrape failures, even though the SSH/host path is perfectly healthy.

Details

The export session is reused across cycles and only reconnected on failure (lib/metrics_target_resource.rb):

def open_resource_session
  return if @session && @last_export_success
  @session = @resource.reload.init_metrics_export_session
  ...
end

export_metrics then does, sequentially on that one session/thread (lib/metrics_target_methods.rb):

def export_metrics(session:, tsdb_client:)
  scrape_results = scrape_endpoints(session)          # SSH exec! (ls/cat)
  ...
  scrape_results.each do |scrape|
    tsdb_client.import_prometheus(scrape, ...)        # HTTP push — can block for seconds
  end
  mark_pending_scrapes_as_done(session, scrape_results[-1].time)  # SSH exec! (rm)
end

The SSH sessions are configured with an aggressive keepalive (model/sshable.rb):

COMMON_SSH_ARGS = {non_interactive: true, timeout: 10,
                   ... keepalive: true, keepalive_interval: 3, keepalive_maxcount: 5}.freeze

i.e. a connection that goes ~15s without a successful keepalive round-trip is considered dead.

Failure mode: when import_prometheus blocks (slow/unavailable backend), the reused SSH session sits idle with no event loop pumping it. The socket goes stale / is reaped (keepalive tolerance, remote sshd idle, or a network middlebox). The next exec! on that session — either mark_pending_scrapes_as_done after the push, or the next cycle's scrape_endpoints — raises IOError: closed stream from net-ssh BasicSocket#send. On top of that, every failed export calls close_resource_session → reconnect next cycle, producing a reconnection storm across the whole target fleet.

Observed impact

⚠️ Correction — see follow-up comment. The causal claim in this section is partly retracted. In the incident that prompted this report, the backend was in fact healthy (flat ingestion, no OOM, no restarts, pushes succeeding) during the window the IOErrors ramped — so they were not caused by slow pushes there, and the "tracked the backend outage / dropped to zero on recovery" statement does not hold for that incident. The structural coupling described above remains a valid latent risk (a genuinely slow/unavailable backend can still produce this signature), it just wasn't the root cause. Original text preserved below for history.

During a metrics-backend outage we saw a large, sustained volume of metrics_export_failure events with IOError: closed stream originating in scrape_endpoints (net-ssh send_pending → BasicSocket#send), alongside the expected push errors (Excon::Error::Timeout, VictoriaMetrics::ClientError, ECONNREFUSED). The IOError rate tracked the backend outage precisely and dropped to zero the instant the backend recovered and pushes became fast again — confirming the SSH failures were a side effect of the slow push holding/idling the shared session, not an SSH or host problem.

The knock-on effect is that buffered metrics stop draining and downstream backlog monitoring fires, making a metrics-sink degradation look like a fleetwide data-plane issue.

Suggested directions

• Decouple the push from the SSH session: finish/close the SSH interaction (scrape into memory) before doing the HTTP import_prometheus, so the session is never held idle across a slow network call.
• And/or validate/refresh the session before reuse (or after a slow import) instead of blindly reusing it.
• And/or bound import_prometheus time and run it off the SSH-holding thread.

Affected files

• lib/metrics_target_methods.rb (export_metrics, scrape_endpoints, mark_pending_scrapes_as_done)
• lib/metrics_target_resource.rb (open_resource_session, export_metrics, close_resource_session)
• model/sshable.rb (COMMON_SSH_ARGS)