Scheduled actions pin the API Lambda ARN, so a deploy that replaces the function strands them

Summary

EventBridge Scheduler pins its target Lambda ARN when a schedule is created. A Pulumi deploy that replaces the API function creates it under a new name and repoints the service manifest, but existing schedules keep the old ARN. They then invoke a Lambda that is stale, or gone.

The blast radius is scheduled CMS publishing and unpublishing. A schedule set for next week survives however many deploys happen in between, and each one can strand it.

How it showed up

While verifying #5657 on a live deployment:

schedule target    : myproj-graphql-9cca423   (last modified 11:14:54 UTC)
manifest lambdaArn : myproj-graphql-0e089bd   (deployed  11:33:55 UTC)

The schedule was created before the 11:33 deploy. That deploy created the API Lambda under a new name and updated the manifest, and the schedule still pointed at the previous one. It fired successfully only because 9cca423 still existed and had also been redeployed recently enough to contain the fix. Neither of those is guaranteed.

Failure modes

Old function still exists. The invocation runs against whatever code that Lambda last had. A schedule created before a release executes against the release before it, so a fix or a behaviour change silently does not apply to already-scheduled actions.

Old function has been removed. The invocation fails. ActionAfterCompletion: "DELETE" removes the schedule after its retries are exhausted, while the ScheduleRecord row stays behind, so Admin keeps showing a Scheduled badge with nothing behind it and the entry never publishes. This is the same silent shape as #5657: the user sees a pending badge forever and no error anywhere.

We already have one of those orphans on the dev instance (wby-schedule-24bc00f9ab7788c7cd094e5a), left by the pre-fix run.

Where it lives

EventBridgeSchedulerService.create() / .update() (packages/api-scheduler-aws/src/features/SchedulerService/EventBridgeSchedulerService.ts) set Target.Arn from this.config.lambdaArn, which comes from the deployed scheduler manifest at the time of the call:

Target: {
    Arn: this.config.lambdaArn,
    RoleArn: this.config.roleArn,
    Input: this.createScheduledActionEventInput(params)
}

Correct at creation. Nothing revisits it when the manifest changes.

Possible directions

Not a recommendation yet, just the shapes worth weighing:

1. Target a stable identifier — a Lambda alias or a versionless ARN that survives replacement, so the pinned target stays valid. Cleanest if Pulumi can guarantee one.
2. Keep the function name stable across deploys and let deploys update code in place, removing the replacement entirely. Widest blast radius, and probably a separate discussion.
3. Re-target on deploy — walk existing schedules after a deploy and point them at the current ARN. Works with today's naming but adds a deploy step that can itself fail.
4. Reconcile at read time — surface a schedule whose target no longer matches the manifest as broken in Admin instead of showing Scheduled. Doesn't fix it, but stops it being silent, which is the worst part.

Whatever we pick, the orphaned-record case deserves fixing on its own: a ScheduleRecord with no live schedule behind it should not render as Scheduled.

Reproducing

1. Schedule a CMS entry to publish some minutes out.
2. Deploy the API and confirm the function name changed (aws lambda list-functions) and the manifest's scheduler.lambdaArn moved with it.
3. Read the schedule's target: it still names the old function.
4. Delete the old function, then wait for the fire time. The entry stays in Draft, the badge stays Scheduled, and nothing is reported.

Found while verifying #5657, but independent of it.