#3703·sarama

[Feature Request] Add a metadata snapshot API

Author: DCjanusCreated Aug 6, 2026Updated Aug 7, 2026

Problem

We operate a large number of Kafka clusters through a centralized management system. To populate its UI, a single service node may scan topic and partition metadata across many clusters. Issuing fresh metadata requests for every scan concentrates network and response-processing costs on that node and can make it a bottleneck.

These scans can tolerate slightly stale data; the important requirement is that they can read the client's existing cache without generating Kafka RPC traffic.

Sarama already maintains this metadata internally, but its public APIs do not provide a complete cache-only view:

  • ClusterAdmin.DescribeTopics sends a MetadataRequest on every call.
  • The Client accessors expose metadata in smaller pieces, and some may refresh metadata on a cache miss.

As a result, there is no way to inspect all currently cached broker and partition metadata with a strict no-refresh guarantee.

Proposal

Expose a concurrency-safe metadata snapshot with these semantics:

  • Reading it never sends a request or triggers a metadata refresh.
  • It represents one consistent state of the client cache.
  • It includes broker addresses and partition metadata such as leader, replicas, ISR, offline replicas, and errors.
  • It is a detached copy, so modifying it cannot affect the client cache.

Callers that need fresh data can explicitly call RefreshMetadata before taking a snapshot.

API compatibility

Our internal fork currently implements this by adding MetadataSnapshot() directly to Client. It has worked well for this use case, but adding a method to a public Go interface breaks external implementations and mocks.

For upstream, a separate optional interface may be a better fit:

go
type MetadataSnapshotter interface {
	MetadataSnapshot() *MetadataSnapshot
}

Sarama's client implementation could implement this interface without changing the existing Client contract.

This is only a suggested API shape. I would be happy to follow maintainer guidance and contribute the implementation.