HIL: concurrent usbtest registration can time out before the battery starts (0/30)

What happened?

Concurrent HIL usbtest batteries intermittently report 0/30 before any case runs. Each battery re-registers the shared cafe:4010 dynamic ID; that registration can block on a healthy peer's active test ioctl and exceed the harness's 15-second timeout. The harness then incorrectly diagnoses a wedged USB subsystem.

This was reproduced on tusb without firmware changes. It explains why the failing board varies with test scheduling while the other examples and peer batteries pass.

Environment

• OS: Linux, 6.12.96+deb13-amd64.
• Harness commit: c1e3bcc354cef3cc270f652a42e8af8bda51b846.
• Firmware: unmodified examples/device/usbtest, copied from CI run 35568107104 at merge commit 76b15de32dfeaccf08c65ca72cbe26fb93318f16; identical artifacts used across the comparison runs.
• Config: test/hil/hfp.json.
• Boards: lpcxpresso43s67, stm32l412nucleo, stm32f746disco, and its DMA variant.

Evidence

The same startup error appears in existing CI logs on different boards:

• LPC failure, run 35130657076.
• STM32F746 failure, run 35568107104.

usbtest did not run: write "cafe 4010 0 0525 a4a0" > /sys/bus/usb/drivers/usbtest/new_id blocked >15s: USB subsystem is wedged (a D-state device lock exists).

Hardware comparison on the same firmware:

The delayed runs deliberately change startup scheduling to model the overlap created by the normal shuffled full suite; they are not unchanged full-suite replays.

In the failing run, the registration writer was sampled blocked for 16.079404 seconds:

tee /sys/bus/usb/drivers/usbtest/new_id
  __driver_attach
  bus_for_each_dev
  usb_store_new_id
  kernfs_fop_write_iter

At the same time, STM32F746 was executing testusb -t 9 -c 1000. Its stack included usbtest_ioctl -> proc_ioctl -> usbdev_ioctl. That case subsequently passed in 20.444359 seconds, and its complete battery passed 30/30. The 5-second stagger caught a shorter, 12.971793-second registration wait, which remained below the timeout and passed.

How to reproduce

On the existing rig, with its tools on PATH and the pinned firmware artifacts available, run one HIL process for the whole set. The harness acquires board locks itself.

For the baseline:

HIL_REPORT_DIR=/tmp/usbtest-baseline \
python3 test/hil/hil_test.py -v -r 1 -t device/usbtest \
  -B /path/to/copied-artifacts test/hil/hfp.json

For the timed reproducer, save this wrapper as staggered-usbtest.py in the repository root:

import sys
import time
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent / 'test' / 'hil'))
import hil_test

original = hil_test.test_device_usbtest

def staggered(board):
    if board['name'] == 'lpcxpresso43s67':
        time.sleep(2)
    return original(board)

hil_test.test_device_usbtest = staggered

if __name__ == '__main__':
    hil_test.main()

Then use the same arguments with the wrapper:

HIL_REPORT_DIR=/tmp/usbtest-staggered \
python3 staggered-usbtest.py -v -r 1 -t device/usbtest \
  -B /path/to/copied-artifacts test/hil/hfp.json

The artifact directory must contain the usual cmake-build-<board> layout, including usbtest and the board_test parking/variant-transition images. The exact timing is rig-dependent; the discriminating condition is registration overlapping a healthy case that keeps the device lock beyond the registration deadline.

Root cause

bind_usbtest() removes and adds the dynamic ID on every battery startup, even when the interface is already bound.

In Linux v6.12.96:

1. usb_store_new_id() calls driver_attach(), scanning devices across the USB bus type, including different host controllers.
2. __driver_attach() acquires the matching interface and parent device locks before the later already-bound check.
3. usbdev_do_ioctl() holds the USB device lock while proc_ioctl() invokes the usbtest driver. A healthy long-running case therefore delays registration for a different board.

The 15-second timeout in sysfs_write() turns this contention into a startup failure. 0/30 here means the battery did not start, not that 30 firmware cases failed.

Proposed fix

Separate host-wide driver preparation from per-board binding:

1. Prepare the known gadget-zero dynamic-ID profile (cafe 4010 0 0525 a4a0) before launching parallel batteries. Per-board startup should reuse that prepared profile and only bind its own interface; it should not repeatedly remove/re-add the shared ID or unnecessarily unbind an already-correct interface.
2. Coordinate preparation/profile replacement with all running batteries on the host, including standalone usbtest.py processes and separate HIL invocations. One possible implementation is a cross-process reader/writer lock: registry/profile changes require exclusive access; active batteries hold shared access. A lock around only new_id writers is insufficient because the competing operation is a peer's test ioctl.
3. Preserve the existing protection against stale or incorrect profiles. Do not register a bare VID/PID without the gadget-zero reference, or assume that an existing ID alone proves the correct profile. Define how initialization is invalidated when the module is reloaded or its registration changes.
4. Change the timeout diagnostic to report a registration timeout/possible lock contention. Declare a permanent wedge only with supporting evidence.

This is a proposed design, not an implemented or validated fix. Increasing the timeout only moves the failure boundary. HIL_USBTEST_PARALLEL=1 is also insufficient as a general fix: its permits are per controller, while registration can wait on devices on another controller. Globally serializing complete batteries is a conservative workaround at the cost of throughput.

Validation for the fix

• Repeat the same staggered reproduction and normal shuffled full-suite runs across reflashes; require 30/30 on every variant.
• Exercise parallel batteries on different controllers and separate processes.
• Cover first registration, already-prepared registration, stale profile correction, module reload, and process exit releasing coordination locks.
• Preserve bounded failure handling for a genuinely stuck device; do not solve contention by allowing unbounded sysfs writes.

The diagnosis left tracked source unchanged, restored all physical boards to verified pristine board_test parking firmware, and released all board locks. Raw stacks, checksums, per-case results and commands are retained on tusb under /home/tusb/hil-runs/usbtest-diagnose-20260921/.