Workflow: opt-in pull-based dispatch for long-running activities

Summary

Add an opt-in activityDispatch setting under spec.workflow in Configuration. In pull mode the Scheduler delivers an activity only to a sidecar with a free execution slot, and that sidecar runs it. Activities stop being pinned to the host their ID hashes to. The default behaviour does not change.

Why

Activities are hashed to a host by placement and cannot move once assigned. With a handful of long-running activities across many replicas, one replica can hold three jobs while others hold none. maxConcurrentActivityInvocations and globalMaxConcurrentActivityInvocations cap how much runs at once but do not move work to idle replicas.

Activities already travel through the Scheduler as jobs, and the Scheduler already gates their delivery on the global and per-name limits. Pull mode reuses that path: the Scheduler chooses a sidecar with capacity, and the receiving sidecar executes the activity instead of forwarding it to the hashed owner. Because the Scheduler holds the waiting work, it can also export a backlog gauge per application and activity name that autoscalers such as KEDA can target. Today no such number exists.

Configuration

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: appconfig
spec:
  workflow:
    maxConcurrentActivityInvocations: 4
    activityDispatch:
      mode: hashed            # hashed (default) | pull
      activities:             # optional per-name overrides, glob patterns
        - name: "Transcode*"
          mode: pull

** Perhaps we want to move the activityDispatch as a member of the per activity concurrency limit instead..?***

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: appconfig
spec:
  workflow:
    activityConcurrencyLimits:
      - name: SendEmail
        maxConcurrent: 5
        dispatchMode: pull

Rules:

• mode: pull requires maxConcurrentActivityInvocations. That value is the number of slots each sidecar offers to the Scheduler. Leaving it unset with pull enabled is a configuration error.
• Per-name entries override mode. Names accept the same glob syntax as WorkflowAccessPolicy.
• globalMaxConcurrentActivityInvocations and activityConcurrencyLimits still apply on top of pull.
• For cross-app activities, the target app's Configuration decides the mode.

Behaviour to be aware of

• Waiting activities are served round-robin across workflow instances, so one large fan-out cannot starve other instances.
• Execution stays at-least-once. If a sidecar dies while running a pull activity, the activity runs again on another sidecar and a late result from the first run is discarded. Expensive activities should already be idempotent; pull mode can make a duplicate run more likely after a crash.
• Enabling pull disables the WorkflowsFastPath preview for that app, the same as configuring global limits does today.
• New Scheduler gauge for autoscaling: scheduler/workflow_activity_backlog tagged with namespace, app_id and activity_name.
• The Helm Configuration CRD must be regenerated with the new fields.