[BUG] Static rate-limit queues can stall after refill or engine restart

Describe the issue

Static rate-limited work does not reliably drain after being durably parked, even with the corrected shouldRefill() predicate discussed in #4346.

I reproduced this against a stock self-hosted v0.101.27 engine using an isolated Docker Compose test environment. Source inspection shows the relevant behavior is unchanged in v0.102.1 and current main (8e33141fb0d291f43a9c2423b88c86b0f635a33d).

Two remaining code paths appear to contribute to this:

1. In pkg/scheduling/v1/queuer.go, RequeueRateLimitedItems is called only when queue-local hasRateLimits is true. That flag is learned from active queue items. Once every remaining item is parked, a newly leased or reconstructed queuer has no active item from which to learn the flag, so it may never call RequeueRateLimitedItems.

2. In pkg/scheduling/v1/rate_limit.go, acknowledged usage is flushed once per second. For a one-second rate limit, usage from the prior window can first reach BulkUpdateRateLimits after the refill boundary. The database then refills the bucket and subtracts the old usage in the same update, consuming capacity from the new window and causing sustained under-admission.

This is separate from the inverted shouldRefill() predicate. The predicate correction is necessary, but it does not fix parked-item rediscovery or the one-second usage-flush boundary.

Environment

• SDK: Go v0.101.27
• Engine: self-hosted v0.101.27
• PostgreSQL and RabbitMQ in an isolated Docker Compose environment
• Also confirmed by source inspection in v0.102.1 and current main at 8e33141fb0d291f43a9c2423b88c86b0f635a33d

Expected behavior

Durably parked tasks should be reconsidered after the relevant rate-limit refill regardless of queue-local history, including after an engine restart. A 1/second key should continue admitting approximately once per second rather than charging usage from the previous window against the newly refilled bucket.

Code to Reproduce, Logs, or Screenshots

Minimal parked-queue reproduction:

1. Create a static 1/minute rate-limit key.
2. Submit six tasks using that key.
3. Allow the first task to start and the remaining tasks to become rate-limit parked.
4. Leave the engine and worker running across at least five complete minute windows.
5. Observe that stock v0.101.27 starts only the first task; the remaining five stay parked.

I ran a wider timestamped matrix. Completed task starts / expected:

Scenario	Stock v0.101.27	With maintenance fix
Fresh 1/minute, five windows	1/6	6/6
Update 5/minute → 1/minute	5/10	10/10
Update 1/minute → 5/minute	1/12	12/12
Update minute → second	5/12	12/12
Fresh 1/second, long horizon	156/180	180/180

The fixed engine also completed the full matrix while workers survived an engine-only restart: 6/6, 10/10, 12/12, 12/12, and 180/180.

I have the disposable Compose reproduction harness, timestamped JSON results, and a small patch that applies cleanly to current main. I can provide the artifacts and open a focused PR after maintainers accept the issue or advise on the preferred design.

Additional context

Related but narrower reports:

• #4346 — inverted shouldRefill() predicate
• #4379 — predicate-only PR

The maintenance fix I tested does not modify shouldRefill(). It instead ensures durable parked queues are examined without relying on queue-local observations, flushes acknowledged usage before the shortest refill boundary, and primes persisted rate-limit state when reconstructing a tenant scheduler.

---

AI Disclosure

• I acknowledge that an LLM was used in the creation of this Issue, in accordance with Hatchet's AI_POLICY.md.

• Details: The Pi coding agent was used for source comparison, development and review of the isolated reproduction harness, analysis of timestamped test results, and the initial draft. GPT-5.6 was used to review and edit the issue text. I encountered the production behavior, directed the investigation and tests, and personally reviewed and edited the diagnosis and report before submission.