#4880·hatchet

[FEAT] Pause a rate-limit key until a timestamp to honor upstream cooldowns

Author: ScalahansoloCreated Sep 3, 2026Updated Sep 16, 2026
Labelsaccepted

Is your feature request related to a problem? Please describe.

Hatchet rate limits meter task starts against a fixed window, which is the right tool for steady-state throughput. They have no way to react to what the upstream actually says. When one task run receives a 429 (or a 503 with Retry-After) from a shared external resource, the engine keeps dispatching every other queued run that consumes the same rate-limit key. Each of those runs starts, immediately fails or bails out, burns a unit, and often makes the upstream cooldown longer. Nothing in the engine can express "this key is closed until 14:32:10."

Teams migrating from other runners expect this. BullMQ has group rate-limit pausing on a rate-limited error; most home-grown runners have some form of "pause this integration for N seconds." Without it, every rate-limited integration on Hatchet ends up with the same bolt-on: an external store (Redis or similar) that records the resume time, a check at the top of every task run, and a durable coordinator that sleeps until the store says go. That is three components, all outside the engine, to implement a single engine-level fact.

Describe the solution you'd like

Let a rate-limit key be paused until a timestamp, from both the API and a task:

typescript
// from a task that just got a 429
await ctx.pauseRateLimit("upstream-api:tenant-123", { until: new Date(Date.now() + retryAfterMs) });

// from outside
await hatchet.ratelimits.pause({ key: "upstream-api:tenant-123", until });
await hatchet.ratelimits.resume({ key: "upstream-api:tenant-123" });

Semantics I would expect:

  • Every queued run whose rendered rate-limit key (static or dynamic) matches stays queued until until, then resumes under the normal window. Runs already executing are unaffected.
  • Pausing is "keep the longest": a later pause with an earlier until does not shorten an existing one. Calls are idempotent.
  • A pause does not consume units, and time spent paused does not count against the run's scheduleTimeout (or, at minimum, this interaction is documented and a per-key option controls it).
  • The pause is visible in ratelimits.list() and the dashboard, with the until timestamp, and it emits a metric alongside hatchet_rate_limited.

Combined with a "retry after" failure outcome on the task itself (#4879), a task that hits an upstream cooldown could pause the shared key and retry itself at the right time in two lines, with no durable wrapper and no external store.

Describe alternatives you've considered

  • ratelimits.upsert({ key, limit: 0 }) and restoring it later. Racy, needs an external timer to restore, and loses the original limit configuration.
  • An external cooldown store checked at the top of each task. Works, but the engine still dispatches runs that then exit immediately, each consuming a unit and a worker slot, and every task author has to remember the check.
  • A durable coordinator per logical job that sleeps until the cooldown ends and spawns a child to do the work. Works, but every sleeping coordinator holds a durable slot and its executionTimeout must cover the cooldown.
  • Setting scheduleTimeout very high so runs wait out the window. Does not help; the window refills on schedule regardless of the upstream's state.

Additional context

  • Rate-limit keys are already a shared engine resource ("the same rendered CEL on multiple steps will be treated as one global rate limit"), so a pause on the key is the natural place for this state.
  • This pairs with #4879 (task-level "retry after" and explicit delay schedules); each is useful on its own, and together they remove the need for the durable-coordinator pattern for rate-limited integrations.

AI Disclosure
  • I acknowledge that an LLM was used in the creation of this Issue, in accordance with Hatchet's AI_POLICY.md.

  • Details: Drafted with Claude Code (Anthropic) after it surveyed the v1 TypeScript SDK 1.30.0 type definitions and docs.hatchet.run. The problem statement comes from a real migration of rate-limited production integrations onto Hatchet v0.105; the text was reviewed by a human before filing.