A transient storage outage permanently fails a job and orphans its continuations

When the storage goes down mid-job, Worker.TryChangeState exhausts its 10 attempts and then forces the FailedState with disableFilters: true (Worker.cs:280; DelayedJobScheduler.cs:382 does the same for Scheduled -> Enqueued). That flag swaps the filtered pipeline for the bare CoreStateMachine, so:

• AutomaticRetryAttribute never converts Failed to Scheduled/Deleted — the job rests in Failed whatever the application's retry policy says (RetryCount is never even set);
• ContinuationsSupportAttribute never runs, so every continuation behind that job is orphaned. FailedState.IsFinal is false, so nothing will ever release them.

Permanent, and needs manual intervention. A job that may have succeeded is recorded as failed and takes an unbounded number of continuations down with it.

Reproduction

Enqueue J1, attach continuations J2..J4 while it is Processing, then alter database [X] set offline with rollback immediate and let J1 throw. Bring the database back once "state change attempts failed" is logged, so the final forced write succeeds:

[Error] Worker: 10 state change attempt(s) failed due to an exception, moving job to the FailedState

head job 1: Failed  ("Failed to change state to a 'Failed' one ... after 10 retry attempts")
awaiting:   2 -> parent 1 (Failed),  3 -> parent 2,  4 -> parent 3

J2..J4 stay Awaiting forever.

Fix

disableFilters exists for a filter that keeps throwing, as the comment in BackgroundJobStateChanger says — but it currently fires for any exception, including a storage outage, where the right action is to do nothing and let the invisibility timeout re-deliver the job.

Calling JobStorage.IsTransientException (the hook #2589 adds) before the fallback and rethrowing when it is true makes the reproduction above recover on its own. Storages that don't override the hook keep today's behaviour. I have a patch with tests and can open it once #2589 lands.

Same category error as #2582: a transient infrastructure failure recorded as a permanent job-level verdict.

The trade-off is re-execution: a job that succeeded but couldn't record it runs again instead of being marked Failed. That falls under the at-least-once contract and beats the current outcome — but it does put weight on classification. An error that is permanent yet reported as transient would re-execute the job indefinitely, where today it would at least come to rest. Worth bearing in mind for #2589's SqlServer override, which treats any DbException as transient — broader than the base method's own documentation advises.

Related: #2582, #2589