[Feature Request] Add a metadata snapshot API
Problem
We operate a large number of Kafka clusters through a centralized management system. To populate its UI, a single service node may scan topic and partition metadata across many clusters. Issuing fresh metadata requests for every scan concentrates network and response-processing costs on that node and can make it a bottleneck.
These scans can tolerate slightly stale data; the important requirement is that they can read the client's existing cache without generating Kafka RPC traffic.
Sarama already maintains this metadata internally, but its public APIs do not provide a complete cache-only view:
ClusterAdmin.DescribeTopicssends aMetadataRequeston every call.- The
Clientaccessors expose metadata in smaller pieces, and some may refresh metadata on a cache miss.
As a result, there is no way to inspect all currently cached broker and partition metadata with a strict no-refresh guarantee.
Proposal
Expose a concurrency-safe metadata snapshot with these semantics:
- Reading it never sends a request or triggers a metadata refresh.
- It represents one consistent state of the client cache.
- It includes broker addresses and partition metadata such as leader, replicas, ISR, offline replicas, and errors.
- It is a detached copy, so modifying it cannot affect the client cache.
Callers that need fresh data can explicitly call RefreshMetadata before taking a snapshot.
API compatibility
Our internal fork currently implements this by adding MetadataSnapshot() directly to Client. It has worked well for this use case, but adding a method to a public Go interface breaks external implementations and mocks.
For upstream, a separate optional interface may be a better fit:
type MetadataSnapshotter interface {
MetadataSnapshot() *MetadataSnapshot
}Sarama's client implementation could implement this interface without changing the existing Client contract.
This is only a suggested API shape. I would be happy to follow maintainer guidance and contribute the implementation.
Source: IBM/sarama