Baike.dev
Log in
Back to tool/Back to issues
#1816·CAP

[Bug] Subscriber `OperationCanceledException` is swallowed and the message is recorded as `Succeeded`

Author: tamazbagdavadzespacegeCreated Sep 15, 2026Updated Sep 21, 2026

Summary

Subscriber OperationCanceledException is swallowed and the message is recorded as Succeeded

Steps to Reproduce

SubscribeExecutor.InvokeConsumerMethodAsync catches OperationCanceledException and ignores it (ISubscribeExector.Default.cs#L198-L201). The caller then runs SetSuccessfulState (#L103). The message is stored as Succeeded with Retries = 0, no exception recorded, an info "executed" log line, and the offset committed (IConsumerRegister.Default.cs#L260). Nothing retries it.

Two ways to hit it:

  1. Shutdown. CAP injects its cancellation token into subscribers that declare a CancellationToken parameter (ISubscribeInvoker.Default.cs#L141). A subscriber that honours it during a restart throws mid-work, and the half-finished message is recorded as Succeeded.
  2. Normal operation. TaskCanceledException derives from OperationCanceledException and is what HttpClient throws on HttpClient.Timeout. A subscriber whose HTTP call times out is recorded as Succeeded, with no CAP cancellation involved.

Found while investigating #. Core library, so all transports and storages are affected. DotNetCore.CAP 10.0.2, master e52b8508.

Root cause

csharp
// ExecuteWithoutRetryAsync
cancellationToken.ThrowIfCancellationRequested();   // L90  guards only the start
await InvokeConsumerMethodAsync(...);               // L99  swallows OCE internally
await SetSuccessfulState(message);                  // L103 runs anyway

// InvokeConsumerMethodAsync
try { var ret = await Invoker.InvokeAsync(ctx, cancellationToken); TracingAfter(...); ... }
catch (OperationCanceledException) { /*ignore*/ }   // L198-201
catch (Exception ex) { TracingError(...); e.ReThrow(); }

Side effects: the retry query only selects Failed/Scheduled rows (IDataStorage.PostgreSql.cs#L349-L350), so a Succeeded row is final. Neither TracingAfter nor TracingError fires, so the OpenTelemetry subscriber span is never stopped (DiagnosticListener.cs#L256-L280). A cancelled callback publish (L194-195, same try) is lost silently.

The catch dates from f664b628 (2019, "Improved Ctrl+C action raised exception issue"). The guards that now serve that purpose live elsewhere: L90 and ISubscribeInvoker.Default.cs#L34 before execution, and Dispatcher catching OperationCanceledException around every executor call (IDispatcher.Default.cs#L132-L135, #L408-L411). The inner swallow only turns failures into successes.

Repro

Fails on e52b8508 (Assert.False() Failure). Same result with a subscriber that takes a CancellationToken, cancels it, and calls ThrowIfCancellationRequested().

csharp
public class HttpTimeoutSubscriber : ICapSubscribe
{
    [CapSubscribe("repro")]
    public Task Handle() => throw new TaskCanceledException("HttpClient.Timeout elapsed", new TimeoutException());
}

[Fact]
public async Task TaskCanceledException_from_subscriber_is_a_failure()
{
    var storage = Substitute.For<IDataStorage>();
    var services = new ServiceCollection();
    services.AddLogging();
    services.AddSingleton<ISerializer, JsonUtf8Serializer>();
    services.AddSingleton<ISubscribeInvoker, SubscribeInvoker>();
    services.AddSingleton(storage);
    services.AddSingleton(Options.Create(new CapOptions()));
    services.AddSingleton<SubscribeExecutor>();
    var executor = services.BuildServiceProvider().GetRequiredService<SubscribeExecutor>();

    var descriptor = new ConsumerExecutorDescriptor
    {
        Attribute = new CandidatesTopic("repro"),
        ServiceTypeInfo = typeof(HttpTimeoutSubscriber).GetTypeInfo(),
        ImplTypeInfo = typeof(HttpTimeoutSubscriber).GetTypeInfo(),
        MethodInfo = typeof(HttpTimeoutSubscriber).GetMethod(nameof(HttpTimeoutSubscriber.Handle))!,
        Parameters = new List<ParameterDescriptor>()
    };
    var message = new MediumMessage
    {
        DbId = "1",
        Origin = new Message(new Dictionary<string, string?>
        {
            [Headers.MessageId] = "1", [Headers.MessageName] = "repro", [Headers.Group] = "g"
        }, null),
        Added = DateTime.Now
    };

    var result = await executor.ExecuteAsync(message, descriptor, CancellationToken.None);

    Assert.False(result.Succeeded);                                                        // actual: true
    await storage.DidNotReceive().ChangeReceiveStateAsync(message, StatusName.Succeeded);  // actual: called once
    Assert.True(message.Retries > 0);                                                      // actual: 0
}

Expected

  1. An OperationCanceledException not caused by CAP's own token is an ordinary failure: SetFailedState, Retries++, retried per FailedRetryCount/FailedRetryInterval.
  2. One caused by CAP's shutdown token must never produce Succeeded. Leave the row recoverable and surface the cancellation to the caller.
  3. TracingError fires so the span closes.

Suggested fix

csharp
catch (OperationCanceledException ex) when (cancellationToken.IsCancellationRequested)
{
    TracingError(tracingTimestamp, message.Origin, descriptor.MethodInfo, ex);
    throw;   // Dispatcher already catches this at L132-135 / L408-411
}
catch (Exception ex)   // now also covers OCE/TaskCanceledException raised by the subscriber itself
{
    var e = new SubscriberExecutionFailedException(ex.Message, ex);
    TracingError(tracingTimestamp, message.Origin, descriptor.MethodInfo, e);
    e.ReThrow();
}

Expected Behavior

No response

Actual Behavior

No response

Log Output

bash

CAP Configuration

yaml

Transport Used

None

Storage Provider

None

Environment

No response

Additional Context

No response

Source: dotnetcore/CAP

View original on GitHubView discussion on GitHub