#3944·tinyusb

HIL: refuse a board still marked wedged instead of paying the pool timeout again

Author: hathachCreated Sep 21, 2026Updated Sep 21, 2026
LabelsPrio 📌followup

A board that stays wedged after a HIL run keeps failing every following run, CI or local, until a human runs usb-kernel-recover. Each of those runs pays the pool guard (HIL_POOL_TIMEOUT, 60 min) to find out.

Today

  • usbtest.py recovers a HUNG case in-run: probe reset first, reflash as fallback (test/hil/usbtest.py:649-700), gated on a convoy-safe flasher (hil_test.py:1558-1578). When that fails, board_wedged latches and the board's remaining examples are skipped.
  • Nothing records the wedge past the run. .github/workflows/build.yml HIL jobs (:446-476, :540-562) upload the report and stop; the next job flashes the wedged board again.
  • hil_report.py has no per-row wedged state; a wedged board is derived from pool-timeout and aborted cells (test/hil/helper/hil_report.py:530).

Remaining work

  • When a wedge survives in-run recovery, hil_test.py writes a per-board unrecovered marker under that board's flock (/tmp/tinyusb-hil-locks/, same reboot semantics as the locks), carrying the run id and the dmesg artifact path.
  • At startup hil_test.py refuses a marked board with a distinct wedged cell, not the locked cell: hil-validate.js retries locked boards automatically and must not retry these.
  • hil_report.py gains a per-row wedged field; the HIL contract's Reporting section and hil-validate.js schema follow deliberately.
  • hil_lock.py gains a subcommand that clears the marker, run only after verified recovery, under the board flock.
  • Limits to document: the marker contains the board's reuse, not the effect of its poisoned usbfs node on other enumerators; a worker killed before writing leaves no marker.

Verification

Unit: marker written on unrecovered wedge, refused at startup, cleared under lock, hil_report.py row shape. Rig (ci.lan): mark a board by hand, run hil_test.py for it, expect a fast wedged row and no flash.

Why deferred

Agreed in the 2026-09-21 HIL review with Codex as the first of five recovery steps; no automatic recovery beyond rung 1 exists yet, so the safe first move is admission, not an always() cleanup step. Later steps: roster flasher_recover for J-Link boards, a shield primitive, post-pool recovery, and the dispatched escalation path.