#3947·tinyusb

HIL: post-pool recovery of wedged boards under the rig-wide hold

Author: hathachCreated Sep 21, 2026Updated Sep 21, 2026
LabelsPrio 📌followup

With the shield primitive (#3946) in place, hil_test.py can recover boards that in-run recovery left wedged, including J-Link boards, before the run ends. The mid-battery HUNG handler is the wrong place: shielding touches the shared root hub, so it must wait for the active batch to drain and hold reservations.

Remaining work

  • A post-pool phase in hil_test.py over the boards latched wedged: reacquire their board flocks (workers close them at hil_test.py:2060-2068), take the rig-wide hold (hil_lock.py hold --all --config <this host's config>) for the shield window, then shield, probe reset, reflash if needed, unshield, and clear the admission marker (#3944) only on verified re-enumeration. Serialized, one board at a time.
  • If the rig-wide reservation is refused, keep the marker and defer; never a partial hold.
  • The flasher runs non-root, or the shield does nothing for it (root ignores the 000 modes). usbtest.py can run as root, so gate on the effective uid, not on sudo -n alone.
  • Update both convoy_safe gates (hil_test.py:1558-1578) and the recovery_reserve() accounting (usbtest.py:60) for the new step; the job timeout-minutes in build.yml stays above the pool guard plus this phase.
  • The existing in-run rung 1 for convoy-safe boards stays as is.
  • The original test verdict is preserved; recovery is reported separately in the report row.

Verification

Rig, ci.lan: deliberate wedge on a J-Link board and an openocd board; interrupted cleanup (kill the phase mid-shield, confirm the stale shield is detected and restored by #3946's ownership record); a concurrent sibling flash on the same root hub during the window; a refused --all keeps the marker. One clean recovery per probe type proves only those cases.

Why deferred

Agreed in the 2026-09-21 HIL review with Codex as step four of five. Depends on #3944 (marker), #3945 (reset timeout) and #3946 (primitive).