indexgateway: Cap client-side in-flight requests to avoid retry amplification when all replicas shed
With -index-gateway.max-concurrent enabled, GatewayClient.poolDo retries a shed request against the next replica, which is the behavior we want: another replica may be less loaded.
When every replica is saturated the same mechanism amplifies load. poolDo passes maxRetries = -1 for six of the seven RPCs, so it walks the entire address set, and in simple mode that set is every discovered gateway. A single request can therefore wait up to max-concurrent-queue-timeout per replica before failing, and each of those waits occupies a queue slot on a gateway that is already overloaded.
A client-side cap on in-flight index gateway requests would let the client fail fast once it can see the whole pool is shedding, instead of serially probing every replica.
Follow-up to https://github.com/grafana/loki/pull/23932.
Source: grafana/loki