Add configurable metric sampling to the Kafka emitter

Description

Add a configurable sampling mechanism to the Kafka emitter community extension. The mechanism should allow operators to selectively drop a percentage of ServiceMetricEvents before they are produced to Kafka.

Suggested configuration properties for KafkaEmitterConfig:

Expected filtering behavior:

1. Only ServiceMetricEvent instances are eligible for sampling.
2. If sampledMetrics is non-empty, sampling applies only when the metric name is present in that set.
3. If sampledNodeTypes is non-empty, sampling applies only when the event's node type is present in that set.
4. For matching events, use pseudo-random selection to retain approximately samplingRate percent of events.
5. Events that do not match the configured metric or node-type filters must be retained.
6. Alert, segment metadata, and request events, when configured, must remain exempt from sampling and continue to be emitted at 100%.
7. Events should be dropped before they reach the Kafka producer buffer so that the feature reduces Kafka produce volume, network traffic, and infrastructure cost.

Motivation

High-traffic Druid deployments can generate millions of metric events. High cardinality metrics such as query/time can create substantial Kafka throughput even when the events are primarily used for trend analysis or anomaly detection.

For example, retaining 10% of per-query latency metrics can provide useful statistical distributions such as p50 and p99 while substantially reducing Kafka ingestion, storage, and network overhead. Consumer-side filtering does not provide the same benefit because the full event volume has already been produced and stored in Kafka.

This feature would let operators choose the trade-off between metric granularity and infrastructure cost while preserving complete delivery of alerts and other critical event types.

Rationale

Sampling at the emitter is preferable to consumer-side filtering because it avoids the cost of serializing, producing, transmitting, and storing events that will eventually be discarded. The proposed matching operation is an O(1) set lookup followed by random selection and should have negligible overhead compared with event serialization.

The default sampling rate of 100 preserves existing behavior for users who do not configure the feature.

Operational impact

This change is opt-in in effect because the default is 100, meaning no events are sampled. Operators using sampling should account for its effect on absolute metric counts. Consumers and BI tools may need to scale sampled metrics when estimating total event volume.

Alerts remain exempt from sampling to avoid reducing monitoring and alerting reliability.

Configuration example

# Basic Kafka emitter configuration
druid.emitter=kafka
druid.emitter.kafka.bootstrap.servers=kafka-broker:9092
druid.emitter.kafka.event.types=["metrics","alerts"]

# Keep 10% of selected high-volume metrics
druid.emitter.kafka.samplingRate=10
druid.emitter.kafka.sampledMetrics=["query/time","query/bytes","sqlQuery/time"]

Test plan

Add coverage for:

• Default configuration retaining all events.
• Sampling rates of 0 and 100.
• Sampling of matching and non-matching metric names.
• Sampling of matching and non-matching node types.
• Empty filter sets applying sampling to all eligible metrics.
• Non-ServiceMetricEvent events always being retained.
• Approximate retention behavior over a sufficiently large event sample.