#6135·opensre

[FEATURE] Scale scheduled-task backlogs and recovery queries

Author: muddlebeeCreated Sep 9, 2026Updated Sep 18, 2026
Labelsenhancementpending triage

Problem statement

Scheduled tasks should remain predictable as task counts and run history grow. PR #6024 bounds active execution, but does not establish large-backlog scalability: admitted callbacks have no explicit queue-size cap, recovery queries scan run history, and the default of two workers is not workload-sized.

For example, if 1,000 distinct tasks become due together, only two execute by default while the rest wait. Sustained arrivals above completion capacity can increase memory use, pending age, and recovery time.

Follow-up to #5963 and #6024. Related bug #6094 tracks worker exhaustion and recovery starvation; this feature tracks capacity planning, backlog visibility, and database performance rather than replacing that bug.

Proposed solution

Keep the existing scheduler and SQLite architecture unless measurements show it is insufficient:

  • Add targeted indexes for live-owner and recovery queries; verify query plans and latency against representative large run histories, including migration of existing databases.
  • Expose pending count and oldest pending age so operators can see overload.
  • Define and document an explicit overload/admission policy that bounds in-memory waiting work without silently discarding accepted runs. Any expiry or coalescing must be an explicit policy, not an incidental effect.
  • Provide measured guidance for OPENSRE_SCHEDULER_MAX_CONCURRENT_RUNS, accounting for task duration, arrival rate, and the shared AI turn gate. Do not simply raise the default.

Success criteria:

  • A repeatable burst and sustained-load test records memory use, queue age, drain/restart recovery time, and recovery-query performance.
  • Query-plan checks demonstrate that completed history does not require a full-table recovery scan.
  • The chosen backlog policy is observable and tested while preserving durable recovery and same-task exclusion.
  • Recovery progress under saturation is verified in coordination with #6094.

No new queue service or distributed scheduler is required by this request.