hil: reduce battery/recovery overhead and simplify lifecycle bookkeeping
Deferred from #3953 (baseline 3c88e5296). A simplify pass over that branch found these behavior-preserving improvements; they touch rig-verified recovery and battery code, so each needs its own measurement or hardware check and would have delayed that PR. Each item lands as its own commit, reviewed, and is accepted or explicitly dropped here with its measurements.
Order: (A+D), C, 2, F, 6, E, 1.
Items
- A+D: build a scan verdict and its evidence together (
test/hil/usbtest.py).wedge_evidence = {'node', 'holders', 'complete'}is built three times besideconfirmation_of(...); replace with_scan_record(node, stuck, complete, **extra)returning(confirmation, evidence)for a completed scan. Keepwaited_s, eachafter, copied holder lists, and the explicit'unverified'assignments before reset/reflash. D: separate the metro_m4 "Keep at least 5 s" note (RECOVER_SETTLE) from the WEDGE_CONFIRM_S comment it runs into. Tests:test_hil_bounded.pyfor cleared / confirmed / incomplete scans, recovery-stage metadata, an exception between invalidation and the next scan. Accept on identical output and cleanup decisions; drop if the helper obscures the state transitions. - C: one not-ran result builder (
hil_report.summarize, built twice). Tests intest_hil_report.py: unknown names matching variants or stale rows, configured boards with no rows, exact fields and distinct details. Stop if it grows into a general result factory. - 2: fetch the report files in one rsync (
.claude/skills/hil/scripts/hil_remote.pycopy_back, three rsyncs today). One rsync with an include filter forhil_report.md,hil_report.jsonand the optional<config>.failedinto a fresh staging dir; success requires exit 0 and both reports staged; then the same all-or-nothing promotion and stale-spec removal. Tests in.claude/test/test_hil_remote.py: pair with and without spec, either report missing, interrupted transfer, stale local files, quoted paths. Accept only with a measured gain (transfer count, wall time). ssh ControlMaster stays out. - F: testusb without the all-device scan. Stock
tools/usb/testusb.cwalks/dev/bus/usbwithftw()and opens and reads every node even with-D;usbdev_open/usbdev_readtakeusb_lock_device(v6.12.107drivers/usb/core/devio.c), so every case start waits for every peer battery's in-flight case. ci.lan journal, 2026-09-22: 6-8 batteries stalled ~20 s and resumed within 20 ms, repeatedly. Make explicit-device mode inspect only its target, keeping interface discovery, the unknown-device fallback,DEVICE,-nand all-device mode. testusb.c is GPL-2.0-or-later: carry a pinned upstream patch plus reproducible rig-build instructions, not the source. First record the rig binary's source revision and hash (/home/hathach/testusb, provenance unknown). Tests: selected-device vs discovery modes, missing or invalid target, interface selection, output compatibility; a trace showing-Dopens no peer node. Rig: repeated full batteries on the same board set, firmware, parameters and concurrency viahil_remote.py, same verdicts, measure case-start delay and wall time. Separate from #3949 (usbtestnew_idregistration). - 6: share process cleanup (
hil_recover._sweepduplicateshil_health._kill_and_confirm;/proc/<pid>/statparsed in three places). First document each caller's identity, grace and failure contract. One byte-based stat parser robust to)and non-ASCII in comm; the smallest shared kill/confirm primitive that keeps recovery's start-time tracking and reparent handling and giveshil_healthpid-reuse protection; caller-specific grace, logging and survivor handling stay.hil_healthtreats an unreadable state as gone today: changing that is a behavior change to call out, not slip in. Tests (deterministic): pid reuse before signalling and during confirmation, reparenting, zombies, ESRCH, EPERM, malformed or unreadable proc data, unknown enumeration, shared deadline, reservation kept while a survivor lives. Rig: a seeded-marker post-run recovery, and an owned identifiable child deliberately left forhil_health's stray cleanup with its termination recorded. Accept on less duplication plus demonstrated identity safety (no speedup needed); if the shared primitive needs too much policy machinery, land only the parser and pid-identity fix. - E: privileged descendants of
usbtest.run(investigate first). On timeoutrun()kills only its direct child, which under sudo is the wrapper; the privileged command survives, and an unprivileged outerkillpgdoes not guarantee a root-owned descendant ends either.run()shares the battery's process group on purpose, so a group kill could hit the caller. Reproduce with a harmless owned sudo child; record pid, start time, ancestry, pgid, privilege and survival. Accept a fix only if owned, killable privileged descendants end within the budget, unresolved survivors stay reported, and unrelated processes and the caller are untouched; stop if ownership cannot be proven. Re-signalling a D-state process alone is no gain. - 1: batch
usb_recover.shattribute operations.attr_opstarts onepython3per sysfs attribute: up to 54 starts per shield (prior estimate ~1.4 s, to remeasure), under the state flock, spent fromSHIELD_TIMEOUT(60 s). Batch while keeping the flock, snapshot before any mutation, record published before chmod,O_PATH/inode checks, rollback, retained records on incomplete restore, owner and overlap checks, compatibility with existing records, the 60 s bound. Tests in.claude/test/test_usb_recover.py: snapshot failure leaves everything untouched, replacement or disappearance races, partial mutation, failed rollback, interrupted run, later unshield, overlap and owner checks. Rig: shield + unshield on one busport underhil_lock.py hold --all --config test/hil/tinyusb.json, then one post-run recovery of a shield-needing board. Accept only with a repeatable gain; the shield now runs only for a non-convoy-safe recovery flasher, so this item may well be dropped.
Verification rules
Affected suites and pre-commit run --all-files per item. On ci.lan, one hil_remote.py invocation per board set; hil_test.py self-locks, never pre-hold its boards. Separate shield or recovery exercises reserve every configured board, respect existing holders and release after verified cleanup; the CI runner stays up. A hardware path not really exercised is reported unverified.
Dropped; do not re-propose without new evidence
hil_testwedge globals merged into one dict: changes representation without simplifying the distinct consumers.- Deriving
wedge_confirmationfrom the last evidence: not equivalent, reset and reflash deliberately invalidate confirmation while keeping earlier evidence. - Boolean wedge state instead of the
WEDGED_*cells: changes the persisted report and its readers for four named states. reserve_allvia a newhil_lock.hold_all(): the CLI owns a detached holder, recovery owns handles in its supervisor.recover_board_attempt()helper: reset and reflash differ in eligibility, budget and result handling.- Inlining or replacing
SETTLE,preconditions(),_alive(): small readable policy boundaries. shield_preconditions()once per phase: checking before each shield keeps failures local.hil_testcosmetics (abortfunctools.partial, chained ternary, redundantisinstance)._owned_rowsvshil_report.variants_of: different fallback semantics on purpose.- ssh ControlMaster for
hil_remote.py. - One post-processing point for
__error__/__survivors__inhil_recover: supervised and unsupervised paths report differently.
Source: hathach/tinyusb