#7978 Support crash-tolerant crawls

JOBDIR only survives a clean stop. On a crash, a double Ctrl-C or a power loss you can lose the whole scheduler queue or corrupt it, lose every in-flight request, and lose or overwrite exported items.

I do not think we should support crash-tolerant crawls by default: crash tolerance means a durable write on every state change plus tracking every in-flight request and item, and most crawls have no reason to pay for that. But it comes up often enough that I think we should make it possible to opt into, with a documented guarantee: after the process dies for any reason, re-running the same job loses no data.

What I think it takes:

Crash-safe scheduler queues. #7877 is the foundation here.
A crash-safe duplicate filter.
Durable scheduler metadata. active.json is only written at close. https://github.com/scrapy/scrapy/pull/8152
Tracking in-flight requests, i.e. those in the downloader or being parsed, so that they are rescheduled on resume.
Tracking in-flight items until every feed they belong to has flushed them.
Resumable feeds: batch IDs surviving a resume, and partially written files being truncated to the last complete record or continued. Maybe limited to specific storages.
A durable spider.state.

I think SQLite is the right disk storage approach in most cases.

In-flight requests and items would be recovered as inputs, not as partial progress: on resume they go through the whole middleware and pipeline chain again, like a retry does. So this requires idempotent item pipelines, and that is on the user. I would rather document that than try to checkpoint mid-chain.

Since enabling it means lining up a matching scheduler, queues, duplicate filter, feed settings and extensions, I would also ship an add-on that configures the whole set, and document that as the way to turn it on.

What I am unsure about:

Which feed storages and formats can be made recoverable at all. Appending to a local .jl file is easy; a single JSON array, a compressed stream or a multipart upload much less so. Some may just be unsupported.
Same for the files and images pipelines: a half-written file in the store is indistinguishable from a complete one.
The durability granularity. One fsync per request is safe and probably unusable; a bounded window of N requests or T seconds is cheaper, but then the guarantee is no longer zero data loss.
How to test this. Killing a real process at many different points seems like the only honest way, and that is a new kind of test for us.
Whether the HTTP cache stays the better answer for people who only want to avoid re-downloading after a crash.

#2399 — resuming after a crash
#4106 — the disk queue is only written on a clean close, and the memory queue never is
#845, #3333, #3413, #1346 — corrupt or wrong-sized requests.queue after an unclean shutdown
#5153 — resuming overwrites %(batch_id)s feed files
#7730 — feed batches only finalized at close
#4749 — a public API for non-graceful stops
#7029 — items from start() are not awaited through pipelines
#500, #3295 — the dbm HTTP cache backend gets corrupted when the spider is killed
#6019 — durability of requests.seen
#4479 — SitemapSpider does not resume
#4326, #4783 — external request queues
#3666 — priority is not honored across the memory and disk queues