Composition pipeline steps have no per-step timeout
What happened?
A composition pipeline has no per-step timeout. Every step's gRPC call to a function inherits the same context, which has a single 2 minute deadline for the whole reconcile (internal/controller/apiextensions/composite/reconciler.go:57):
timeout = 2 * time.Minute
...
ctx, cancel := context.WithTimeout(ctx, timeout)PackagedFunctionRunner.RunFunction (internal/xfn/function_runner.go:158) just forwards this same context to the gRPC call with no per-call deadline override.
If one step in a pipeline is slow (e.g. a function calling an external API, or a function pod under load), it can consume the entire 2 minute budget by itself. Steps after it never run in that reconcile. There is no way to bound how long a single step is allowed to take.
Why this matters
- A pipeline with 5 fast steps and 1 slow step behaves the same as a pipeline that is entirely slow. There's no isolation between steps.
- A function that hangs (not crashes, just hangs) blocks the whole reconcile until the 2 minute deadline, instead of failing fast on the one bad step.
- There's no way for a Composition author to say "this step talks to an external system, give it more time" or "this step should be fast, fail it quickly if it isn't."
- Debugging is harder: when the reconcile times out, the error doesn't say which step was actually running when the deadline hit.
How could Crossplane help solve this?
Add an optional per-step timeout to the pipeline step API, e.g.:
pipeline:
- step: call-external-api
functionRef:
name: function-foo
timeout: 30sIf unset, fall back to current behavior (shared with the overall reconcile timeout). When a step's individual timeout is hit, surface a clear error naming the step, instead of a generic deadline-exceeded error after the full reconcile budget is gone.
This would need:
- A new optional field on the pipeline step type in the Composition API
composition_functions.goderiving a child context withcontext.WithTimeoutper step when the field is set- A clear error message identifying which step timed out, similar to how
SEVERITY_FATALalready names the step (PipelineFatalErrorFmt)
Source: crossplane/crossplane