Baike.dev
All toolsAI codingTrendingOpen sourceNewsSubmit
Log in
Back to tool/Back to issues
#24311·loki

Distributor writes return 5xx during graceful ingester scale-down (no write extension)

Author: 17billionCreated Aug 31, 2026Updated Sep 14, 2026
Labelstype/featurecomponent/distributor

Is your feature request related to a problem? Please describe.

The distributor builds the ingester write set with ring.WriteNoExtend. When an ingester is briefly non-ACTIVE, for example LEAVING during a graceful scale-down or a rollout, it is dropped from the replica set with no replacement. If there is no quorum slack (RF=2, or RF=3 when several ingesters change at once), ring.Get returns at least N live replicas required and the write fails with 5xx, even though the shutdown was graceful.

With ingester autoscaling this shows up as recurring write 5xx and constant retries from clients (Promtail, Fluentd, and so on) during scale-down and rollouts.

Describe the solution you'd like

Let the distributor use ring.Write, which replaces a LEAVING or JOINING ingester with the next healthy one and keeps write quorum while the ring changes, behind an opt-in flag that defaults to off. Mimir already exposes -distributor.extend-writes for the same purpose. Reads already include LEAVING instances and extend, so anything written to the extension target is still queried.

#23908 implements this.

Describe alternatives you've considered

Running with more quorum slack (higher replication factor) avoids the failures but costs more, and does not help when several ingesters change at once.

Source: grafana/loki

View original on GitHubView discussion on GitHub