(meta-bug?) Better rate-limiting

We wanted to re-think what we want/need about per-request delays, retrying 429 after a set delay etc. There are several related issues and PRs, I'm trying to collect them here.

• https://github.com/scrapy/scrapy/issues/802 (and its open PRs https://github.com/scrapy/scrapy/pull/4980 and https://github.com/scrapy/scrapy/pull/5015)
• https://github.com/scrapy/scrapy/issues/892 (not necessarily related to retrying/throttling)
• https://github.com/scrapy/scrapy/issues/3849
• https://github.com/scrapy/scrapy/issues/4424 (and its open PR https://github.com/scrapy/scrapy/pull/4462)
• https://github.com/scrapy/scrapy/issues/6476 - somewhat related
• https://github.com/scrapy/scrapy/issues/6597 - also somewhat related
• Ability to set a minimum RPS/RPM, and have concurrency automatically increased temporarily as needed to maintain it.
• https://github.com/scrapy/scrapy/issues/157 - limiting bandwidth per server
• https://github.com/scrapy/scrapy/issues/305 - website-specific way to join multiple requests and send them together - it may make more sense to close it though