Interrupted sync executions are invisible in metrics — a checkpoint-stalled sync loops forever while reporting success
TL;DR: When a sync execution is interrupted at the duration cap, it is recorded as a success everywhere except one log payload field. A sync whose checkpoint never advances (the "never finishes" state warned about in the checkpoints docs) is therefore indistinguishable — in metrics, task status, and log message text — from a healthy sync that runs frequently. It burns the customer's upstream API quota and Nango compute indefinitely, and neither side can see it.
The chain (file:line refs from master)
- Lambda executions get
interruptAfterMs—packages/jobs/lib/execution/sync.ts:215 - Graceful interruption produces
error.type === 'execution_interrupted', which is converted to success:handleSyncSuccess({..., interrupted: true})—packages/jobs/lib/execution/operations/handler.ts:59-61 - In
handleSyncSuccess:interruptedgoes into the operation log payload only (logCtx.enrichOperationmeta)- the log message text reads "…completed successfully" either way —
sync.ts:~520 updateSyncJobStatus(SUCCESS)andsetTaskSuccess— counted as successnextExecutionInMs: 0— immediate re-run —sync.ts:541
- No metric anywhere carries
interrupted.FUNCTION_EXECUTIONS(packages/metering/lib/processors/usage.ts:236) has dimensions{type, success, accountId, frequencyBucket, functionRuntime}— success istruefor interrupted runs.TASKS_FAILED/TASKS_EXPIREDnever fire.
Why routing doesn't prevent it
packages/jobs/lib/runtime/runtimes.rules.ts:29 keeps checkpoint-less syncs off the Lambda
fleet — but only when plan.sync_lambda_checkpoint_required is set, and only by checking
whether the sync declares the checkpoints feature. A sync that declares a checkpoint
schema but whose checkpoint does not advance (stalled cursor, buggy customer code saving the
same value, dataset growing faster than the cap allows) passes routing and loops forever.
Capability is checked; progress never is.
The fix is cheap because the data is already in hand
handleSyncSuccess already receives both interrupted and checkpoints: CheckpointRange
({from, to} — packages/types/lib/checkpoint/types.ts:22). Detecting the pathological
state is a comparison of two values already in scope:
--- a/packages/utils/lib/telemetry/metrics.ts
+++ b/packages/utils/lib/telemetry/metrics.ts
@@ enum Types
+ SYNC_EXECUTION_INTERRUPTED = 'nango.sync.execution.interrupted',
--- a/packages/jobs/lib/execution/sync.ts
+++ b/packages/jobs/lib/execution/sync.ts
@@ -32,1 +32,1
-import { Err, getFrequencyMs, Ok, tagTraceUser } from '@nangohq/utils';
+import { Err, getFrequencyMs, metrics, Ok, tagTraceUser } from '@nangohq/utils';
@@ handleSyncSuccess, before setTaskSuccess
+ if (interrupted) {
+ const checkpointAdvanced =
+ !!checkpoints && JSON.stringify(checkpoints.from) !== JSON.stringify(checkpoints.to);
+ metrics.increment(metrics.Types.SYNC_EXECUTION_INTERRUPTED, 1, {
+ checkpointProgress: checkpointAdvanced ? 'advanced' : 'none'
+ });
+ if (!checkpointAdvanced) {
+ void logCtx.warn(
+ `Sync '${nangoProps.syncConfig.sync_name}' was interrupted at the execution ` +
+ `duration cap without saving checkpoint progress. If this repeats on every run, ` +
+ `the sync restarts from scratch each time and will never complete. Save a ` +
+ `checkpoint after every batchSave: ` +
+ `https://nango.dev/docs/guides/functions/syncs/checkpoints`
+ );
+ }
+ }Cardinality: one new metric, one static dimension with two values — consistent with the
gated-cardinality approach in CARDINALITY_GATED_PROVIDER_CONFIG_KEY_METRICS. No per-account
dimension (per-account views already exist via FUNCTION_EXECUTIONS).
The logCtx.warn half also makes the failure visible to the customer in their own
operation logs — today the docs warn about this state, but nothing in the product ever
tells the person whose sync is stuck in it.
An alert of the form interrupted{checkpointProgress="none"} repeated N times for the same sync then makes "will never complete" a detectable condition instead of a documented hazard.
Happy to expand on any of this. (Aware external PRs aren't reviewed per CONTRIBUTING.md — the diff is here so the change is one cherry-pick away if you want it.)
Source: NangoHQ/nango