Feature Request: autoscaling for Kafka share groups (KIP-932 queues)

Proposal

Add first-class support for autoscaling Kafka share-group consumers (KIP-932, "Queues for Kafka"). The existing Apache Kafka scaler is built on committed consumer-group offsets and is partition-bound, which is structurally incompatible with share groups. We'd like a scaler (or a mode of the existing one — open to maintainers' preference) that scales on share-group backlog.

Use-Case

KIP-932 share groups let many consumers cooperatively read the same partition, so consumer parallelism is no longer capped by partition count. That's exactly why teams adopt them — to scale workers without over-partitioning (and without adding brokers). Today KEDA can't drive that elasticity for a share group, so adopters fall back to exporting broker metrics to Prometheus and using the prometheus scaler (see Confluent's queues-for kafka demo), which is bespoke and fragile.

Is this a feature you are interested in implementing yourself?

No

Anything else?

Why the current Kafka scaler doesn't fit

• It computes lag = log-end-offset − committed-offset per partition for a consumer group. Share groups don't track progress as a single committed offset per partition; the broker tracks per-record delivery state (acquired /acknowledged / released / archived) plus a share-partition start offset. There is no committed offset to subtract.
• It caps replicas at the partition count (allowIdleConsumers to override). For share groups that cap is the opposite of what you want — the point is >1 consumer per partition.

Proposed scaling signal: share-group backlog

Scale on the number of records available to be delivered but not yet acknowledged for a given share group / topic - the share-group analogue of lag. Candidate sources (maintainers to confirm the cleanest against the shipped APIs):

• Share-group admin surface - DescribeShareGroups / list-share-group-offsets (kafka-share-groups.sh): share-partition start offset vs. log-end-offset gives an approximate backlog, augmented by in-flight / released counts.
• Broker-side share-group metrics (unacked / in-flight / redelivery counts).

Prior references

• KIP-932: Queues for Kafka (Apache cwiki)
• Confluent queues-for-kafka-demo - KEDA-via-Prometheus workaround
• Related existing issue #5096 (scale-down/redistribution; not share-group specific)