CronOperation stuck in infinite "already exists" loop when API server creationTimestamp is 1s behind scheduled time

What happened?

A CronOperation enters an infinite error loop, repeatedly failing to create an Operation that already exists. The CronOperation never recovers (Synced: False) and stops scheduling new Operations until the conflicting Operation is manually deleted.

Warning  CreateOperation  73s (x143 over 136m)  ops/cronoperation.ops.crossplane.io
  cannot create scheduled Operation "my-cronop-1781709000":
  operations.ops.crossplane.io "my-cronop-1781709000" already exists

Expected behavior: The CronOperation should continue scheduling new Operations on subsequent ticks, even if an Operation for a given schedule slot already exists.

Root cause: The CronOperation controller uses two different time sources that can disagree by ≥1 second:

1. Operation name uses the scheduled time: fmt.Sprintf("%s-%d", co.GetName(), scheduled.Unix())
2. LastScheduleTime is derived from the K8s creationTimestamp of the most recent Operation via lifecycle.LatestCreateTime()

When there's ≥1 second of clock skew between the controller pod and the API server (common in any multi-node cluster), creationTimestamp lands just before the schedule boundary. On subsequent reconciles, Next(schedule, creationTimestamp) computes the same scheduled time as the existing Operation, and the Create fails with AlreadyExists. The controller does not handle this error — it returns it and requeues, creating an infinite loop.

Relevant code in internal/controller/ops/cronoperation/reconciler.go:

• Line ~115: LastScheduleTime unconditionally overwritten from LatestCreateTime() (K8s creationTimestamp) on every reconcile
• Line ~148: Next(schedule, last) recomputes the same timestamp as the existing Operation
• Line ~213: Create() error returned without checking for AlreadyExists

Detailed reconciliation flow

Tick at 15:10:00 — Operation created with clock skew:

1. Controller clock: 15:10:00.050, computes next = 15:10:00
2. Creates my-cronop-1781709000 successfully
3. API server stamps creationTimestamp: 15:09:59Z (1s behind)

Next tick at 15:15:00 — Loop begins:

1. Lists Operations, finds my-cronop-1781709000 with creationTimestamp: 15:09:59
2. LatestCreateTime() → 15:09:59
3. Overwrites LastScheduleTime = 15:09:59 (unconditional, every reconcile)
4. Next("*/5 * * * *", 15:09:59) → 15:10:00
5. Tries to create my-cronop-1781709000 → AlreadyExists
6. Returns error, requeues, repeats forever

The key issue: LastScheduleTime is unconditionally overwritten from LatestCreateTime() on every reconcile, making the 1-second skew permanent. The controller can never advance past it.

How can we reproduce it?

This triggers any time the K8s API server's clock is ≥1 second behind the Crossplane controller pod's clock at the moment an Operation is created. The API server stamps creationTimestamp using its own clock, which may be slightly behind the controller's time.Now() that determined it was time to schedule.

Once triggered, the CronOperation stays stuck indefinitely until the conflicting Operation is manually deleted.

Standalone Go program demonstrating the issue:

package main

import (
	"fmt"
	"time"
	"github.com/robfig/cron/v3"
)

func main() {
	schedule := "*/5 * * * *"
	cs, _ := cron.ParseStandard(schedule)

	// Operation was scheduled for 15:10:00 but K8s stamped it 15:09:59
	scheduledTime := time.Date(2026, 6, 17, 15, 10, 0, 0, time.UTC)
	creationTimestamp := time.Date(2026, 6, 17, 15, 9, 59, 0, time.UTC)

	fmt.Printf("Operation name: my-cronop-%d (from scheduled time)\n", scheduledTime.Unix())
	fmt.Printf("creationTimestamp: %s\n\n", creationTimestamp)

	// Controller re-derives LastScheduleTime from creationTimestamp on every reconcile
	lastScheduleTime := creationTimestamp
	next := cs.Next(lastScheduleTime)
	fmt.Printf("Next(%q, %s) = %s\n", schedule, lastScheduleTime, next)
	fmt.Printf("Tries to create: my-cronop-%d -- ALREADY EXISTS, loops forever\n", next.Unix())
}

Output:

Operation name: my-cronop-1781709000 (from scheduled time)
creationTimestamp: 2026-06-17 15:09:59 +0000 UTC

Next("*/5 * * * *", 2026-06-17 15:09:59 +0000 UTC) = 2026-06-17 15:10:00 +0000 UTC
Tries to create: my-cronop-1781709000 -- ALREADY EXISTS, loops forever

Minimal CronOperation manifest to reproduce:

apiVersion: ops.crossplane.io/v1alpha1
kind: CronOperation
metadata:
  name: my-cronop
spec:
  schedule: "*/5 * * * *"
  concurrencyPolicy: Forbid
  successfulHistoryLimit: 3
  failedHistoryLimit: 3
  operationTemplate:
    spec:
      mode: Pipeline
      pipeline:
        - step: my-step
          functionRef:
            name: function-auto-ready

Workaround: Manually delete the conflicting Operation:

kubectl delete operation <cronoperation-name>-<unix-timestamp>

Suggested fix

Derive LastScheduleTime from the scheduled time (Operation name) instead of creationTimestamp

func LatestScheduledTime(cronName string, ops ...v1alpha1.Operation) time.Time {
    prefix := cronName + "-"
    var latest time.Time
    for _, op := range ops {
        name := op.GetName()
        if !strings.HasPrefix(name, prefix) {
            continue
        }
        unix, err := strconv.ParseInt(strings.TrimPrefix(name, prefix), 10, 64)
        if err != nil {
            continue
        }
        if t := time.Unix(unix, 0); t.After(latest) {
            latest = t
        }
    }
    return latest
}

What environment did it happen in?

Crossplane version: v2.1.3

• Cloud provider: AWS (EKS)
• Kubernetes version: 1.33
• Kubernetes distribution: Amazon EKS