#3360·metaflow

MetadataHeartBeat._heartbeat posts without a timeout, so its own `except requests.exceptions.Timeout` handler is not reachable

Author: RudraDudhat2509Created Sep 2, 2026Updated Sep 2, 2026

_heartbeat() in metaflow/metadata_provider/heartbeat.py does a requests.post with no timeout, but there's an except requests.exceptions.Timeout handler right below it. requests doesn't time out by default, so that handler can never actually fire.

_ping() above it retries by catching HeartBeatException:

python
except HeartBeatException as e:
    retry_counter = retry_counter + 1
    time.sleep(1.5**retry_counter)

So if the metadata service accepts the connection but stops responding, _heartbeat() never returns and never raises, and that retry never runs. The heartbeat thread just sits there and the run stops heartbeating without anything showing up.

Reproduced with a socket that accepts and never replies, then the same post heartbeat.py makes today:

python
import socket, threading, requests

srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.bind(("127.0.0.1", 0)); srv.listen(5)
port = srv.getsockname()[1]

def stall():
    conns = []
    while True:
        c, _ = srv.accept(); conns.append(c)   # hold open, never respond

threading.Thread(target=stall, daemon=True).start()
requests.post(url=f"http://127.0.0.1:{port}/ping", data="{}", headers={})

Still blocked past 15s. With timeout=(3.05, 10) it raises Timeout after 10s, so the existing handler fires.

Happy to open a pr with a test that fails without the fix. Not the same as #2924, that one is the aws sandbox sts call in aws_client.py. alr working on a fix on local, should push soon