Rate limiting: Add global exponential back off duration configuration

What's happening?

Sometimes when generating large number of beams or merges in parallel, I can hit server rate limits.

A 300,000 tokens request, sent to 10 beams exceeds 2,000,000 input tokens per minute typical Anthropic limit.

One of the solutions I proposed was to add an option to specify rate limit values: https://github.com/enricoros/big-AGI/issues/979

However, it is potentially fragile for a variety of reasons (different providers handle rate limits differently, some have separate input/output limits, some have combined, some are more and some are less strict on the exact limit).

An alternative solution is to implement exponential back-off with user configurable maximum duration of retrying, so that I can set, for example, a 2 minute limit for retires based on my typical workloads, and still have the retry process end and indicate an error if something unusual is happening.

Right now I have to manually click buttons to resend beams or merges that failed due to rate limit errors.

All major providers return HTTP 429 ("Too Many Requests") code on rate limit errors, so it is easy to distinguish it from other issues.

A good explanation of exponential backoff with jitter is provided at https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/ and https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/

Where does this happen?

Big-AGI Pro (big-agi.com)

Impact on your workflow

Medium - Workaround exists

Environment (if applicable)

No response

Additional context

No response