HIL: refuse a board still marked wedged instead of paying the pool timeout again
A board that stays wedged after a HIL run keeps failing every following run, CI or local, until a human runs usb-kernel-recover. Each of those runs pays the pool guard (HIL_POOL_TIMEOUT, 60 min) to find out.
Today
usbtest.pyrecovers a HUNG case in-run: probe reset first, reflash as fallback (test/hil/usbtest.py:649-700), gated on a convoy-safe flasher (hil_test.py:1558-1578). When that fails,board_wedgedlatches and the board's remaining examples are skipped.- Nothing records the wedge past the run.
.github/workflows/build.ymlHIL jobs (:446-476,:540-562) upload the report and stop; the next job flashes the wedged board again. hil_report.pyhas no per-row wedged state; awedgedboard is derived from pool-timeout and aborted cells (test/hil/helper/hil_report.py:530).
Remaining work
- When a wedge survives in-run recovery,
hil_test.pywrites a per-board unrecovered marker under that board's flock (/tmp/tinyusb-hil-locks/, same reboot semantics as the locks), carrying the run id and the dmesg artifact path. - At startup
hil_test.pyrefuses a marked board with a distinct wedged cell, not the locked cell:hil-validate.jsretrieslockedboards automatically and must not retry these. hil_report.pygains a per-rowwedgedfield; the HIL contract's Reporting section andhil-validate.jsschema follow deliberately.hil_lock.pygains a subcommand that clears the marker, run only after verified recovery, under the board flock.- Limits to document: the marker contains the board's reuse, not the effect of its poisoned usbfs node on other enumerators; a worker killed before writing leaves no marker.
Verification
Unit: marker written on unrecovered wedge, refused at startup, cleared under lock, hil_report.py row shape. Rig (ci.lan): mark a board by hand, run hil_test.py for it, expect a fast wedged row and no flash.
Why deferred
Agreed in the 2026-09-21 HIL review with Codex as the first of five recovery steps; no automatic recovery beyond rung 1 exists yet, so the safe first move is admission, not an always() cleanup step. Later steps: roster flasher_recover for J-Link boards, a shield primitive, post-pool recovery, and the dispatched escalation path.
Source: hathach/tinyusb