Task processor: one slow operator task freezes every group of its application

A single operator task has no bound on its wall-clock time, and the blast radius of one slow task is the whole application rather than its own task group.

The contract we already document

ProcessorActions::execute_tasks (linera-base/src/task_processor.rs) states:

The outcomes of distinct groups commute: each is submitted as soon as its task succeeds, in no guaranteed order relative to the other groups, and a task that fails is retried without holding the other groups back.

That holds for a group that fails quickly. It does not hold for one that is slow or hung.

Where it breaks

In linera-service/src/task_processor.rs:

1. in_flight_apps: BTreeSet<ApplicationId> (line 69) is keyed by application. process_actions skips the entire application while any batch is in flight (line 207), so no group of that application can be polled or started while a sibling group is still running.
2. The spawned batch awaits every group handle before sending its BatchResult (lines ~240-265), and retry_at is the max over all groups. The application is unblocked only by the slowest group.
3. execute_task never bounds the child — child.wait_with_output() (line 351) waits indefinitely, and there is no CLI knob for it. --task-retry-delay-secs paces retries between attempts; nothing caps one attempt.

Together: one operator process that does not exit freezes every consumer of that application for as long as it runs.

Observed

oracle-v2-71-1 on 2026-08-18. Two consumers of the same oracle application, HYPE-1-lv and ETH-1-lv:

21:29:01  ETH   submits ts 1787088540
          HYPE  group still running; its operator child never returns
21:30-21:43  no log line mentioning the chain at all
21:42:03  HYPE  finally returns ts 1787088540 — the same timestamp
21:42:04  ETH   catches up 13 points (21:30 -> 21:42)

Both consumers stuck on the identical timestamp is what identifies them as one batch. ETH-1-lv is a dense, perfectly healthy symbol; it lost 13 minutes of block production waiting on an unrelated group. Nothing failed loudly — the pod stayed Ready, so no liveness probe fired.

Proposed

1. Bound a task's wall clock. Wrap execute_task's child in tokio::time::timeout with Command::kill_on_drop(true) so the timeout actually reaps the process, and expose the duration as a CLI argument alongside --task-retry-delay-secs. This alone turns "hung forever" into "fails after T", which the existing per-group retry logic already handles correctly.
2. Make in-flight tracking per group. Key it by (ApplicationId, Option<String>) and let each group report its own result, so a slow group no longer gates its siblings' polling or their retry scheduling. This is what makes the documented contract true rather than nearly true.

(2) is safe with respect to cursors under the assumption the framework already documents — "Tasks are assumed idempotent, so whatever is left unsubmitted is recomputed by the next call to nextActions" — so re-polling an application while one of its groups is in flight re-returns that group's outstanding work and the in-flight filter drops the duplicate.

Notes

• The related ordering coupling was fixed by #6697/#6698 (Task.id groups, submitted independently). That change is live and works: during a burst where one group failed ~40 times in a row, its siblings kept publishing on schedule. It decoupled failure ordering, not duration.
• Detection gap worth tracking separately: the process stays healthy and the pod stays Ready throughout, so this is invisible to liveness probes.