#3954·tinyusb

hil: reduce battery/recovery overhead and simplify lifecycle bookkeeping

Author: hathachCreated Sep 22, 2026Updated Sep 22, 2026
LabelsPrio 📌followup

Deferred from #3953 (baseline 3c88e5296). A simplify pass over that branch found these behavior-preserving improvements; they touch rig-verified recovery and battery code, so each needs its own measurement or hardware check and would have delayed that PR. Each item lands as its own commit, reviewed, and is accepted or explicitly dropped here with its measurements.

Order: (A+D), C, 2, F, 6, E, 1.

Items

  • A+D: build a scan verdict and its evidence together (test/hil/usbtest.py). wedge_evidence = {'node', 'holders', 'complete'} is built three times beside confirmation_of(...); replace with _scan_record(node, stuck, complete, **extra) returning (confirmation, evidence) for a completed scan. Keep waited_s, each after, copied holder lists, and the explicit 'unverified' assignments before reset/reflash. D: separate the metro_m4 "Keep at least 5 s" note (RECOVER_SETTLE) from the WEDGE_CONFIRM_S comment it runs into. Tests: test_hil_bounded.py for cleared / confirmed / incomplete scans, recovery-stage metadata, an exception between invalidation and the next scan. Accept on identical output and cleanup decisions; drop if the helper obscures the state transitions.
  • C: one not-ran result builder (hil_report.summarize, built twice). Tests in test_hil_report.py: unknown names matching variants or stale rows, configured boards with no rows, exact fields and distinct details. Stop if it grows into a general result factory.
  • 2: fetch the report files in one rsync (.claude/skills/hil/scripts/hil_remote.py copy_back, three rsyncs today). One rsync with an include filter for hil_report.md, hil_report.json and the optional <config>.failed into a fresh staging dir; success requires exit 0 and both reports staged; then the same all-or-nothing promotion and stale-spec removal. Tests in .claude/test/test_hil_remote.py: pair with and without spec, either report missing, interrupted transfer, stale local files, quoted paths. Accept only with a measured gain (transfer count, wall time). ssh ControlMaster stays out.
  • F: testusb without the all-device scan. Stock tools/usb/testusb.c walks /dev/bus/usb with ftw() and opens and reads every node even with -D; usbdev_open/usbdev_read take usb_lock_device (v6.12.107 drivers/usb/core/devio.c), so every case start waits for every peer battery's in-flight case. ci.lan journal, 2026-09-22: 6-8 batteries stalled ~20 s and resumed within 20 ms, repeatedly. Make explicit-device mode inspect only its target, keeping interface discovery, the unknown-device fallback, DEVICE, -n and all-device mode. testusb.c is GPL-2.0-or-later: carry a pinned upstream patch plus reproducible rig-build instructions, not the source. First record the rig binary's source revision and hash (/home/hathach/testusb, provenance unknown). Tests: selected-device vs discovery modes, missing or invalid target, interface selection, output compatibility; a trace showing -D opens no peer node. Rig: repeated full batteries on the same board set, firmware, parameters and concurrency via hil_remote.py, same verdicts, measure case-start delay and wall time. Separate from #3949 (usbtest new_id registration).
  • 6: share process cleanup (hil_recover._sweep duplicates hil_health._kill_and_confirm; /proc/<pid>/stat parsed in three places). First document each caller's identity, grace and failure contract. One byte-based stat parser robust to ) and non-ASCII in comm; the smallest shared kill/confirm primitive that keeps recovery's start-time tracking and reparent handling and gives hil_health pid-reuse protection; caller-specific grace, logging and survivor handling stay. hil_health treats an unreadable state as gone today: changing that is a behavior change to call out, not slip in. Tests (deterministic): pid reuse before signalling and during confirmation, reparenting, zombies, ESRCH, EPERM, malformed or unreadable proc data, unknown enumeration, shared deadline, reservation kept while a survivor lives. Rig: a seeded-marker post-run recovery, and an owned identifiable child deliberately left for hil_health's stray cleanup with its termination recorded. Accept on less duplication plus demonstrated identity safety (no speedup needed); if the shared primitive needs too much policy machinery, land only the parser and pid-identity fix.
  • E: privileged descendants of usbtest.run (investigate first). On timeout run() kills only its direct child, which under sudo is the wrapper; the privileged command survives, and an unprivileged outer killpg does not guarantee a root-owned descendant ends either. run() shares the battery's process group on purpose, so a group kill could hit the caller. Reproduce with a harmless owned sudo child; record pid, start time, ancestry, pgid, privilege and survival. Accept a fix only if owned, killable privileged descendants end within the budget, unresolved survivors stay reported, and unrelated processes and the caller are untouched; stop if ownership cannot be proven. Re-signalling a D-state process alone is no gain.
  • 1: batch usb_recover.sh attribute operations. attr_op starts one python3 per sysfs attribute: up to 54 starts per shield (prior estimate ~1.4 s, to remeasure), under the state flock, spent from SHIELD_TIMEOUT (60 s). Batch while keeping the flock, snapshot before any mutation, record published before chmod, O_PATH/inode checks, rollback, retained records on incomplete restore, owner and overlap checks, compatibility with existing records, the 60 s bound. Tests in .claude/test/test_usb_recover.py: snapshot failure leaves everything untouched, replacement or disappearance races, partial mutation, failed rollback, interrupted run, later unshield, overlap and owner checks. Rig: shield + unshield on one busport under hil_lock.py hold --all --config test/hil/tinyusb.json, then one post-run recovery of a shield-needing board. Accept only with a repeatable gain; the shield now runs only for a non-convoy-safe recovery flasher, so this item may well be dropped.

Verification rules

Affected suites and pre-commit run --all-files per item. On ci.lan, one hil_remote.py invocation per board set; hil_test.py self-locks, never pre-hold its boards. Separate shield or recovery exercises reserve every configured board, respect existing holders and release after verified cleanup; the CI runner stays up. A hardware path not really exercised is reported unverified.

Dropped; do not re-propose without new evidence

  • hil_test wedge globals merged into one dict: changes representation without simplifying the distinct consumers.
  • Deriving wedge_confirmation from the last evidence: not equivalent, reset and reflash deliberately invalidate confirmation while keeping earlier evidence.
  • Boolean wedge state instead of the WEDGED_* cells: changes the persisted report and its readers for four named states.
  • reserve_all via a new hil_lock.hold_all(): the CLI owns a detached holder, recovery owns handles in its supervisor.
  • recover_board _attempt() helper: reset and reflash differ in eligibility, budget and result handling.
  • Inlining or replacing SETTLE, preconditions(), _alive(): small readable policy boundaries.
  • shield_preconditions() once per phase: checking before each shield keeps failures local.
  • hil_test cosmetics (abort functools.partial, chained ternary, redundant isinstance).
  • _owned_rows vs hil_report.variants_of: different fallback semantics on purpose.
  • ssh ControlMaster for hil_remote.py.
  • One post-processing point for __error__/__survivors__ in hil_recover: supervised and unsupervised paths report differently.