veneur

Table of Contents

• What Is Veneur?
  • Use Case
  • See Also
• Status
• Features
  • Vendor And Backend Agnostic
  • Modern Metrics Format (Or Others!)
  • Global Aggregation
  • Approximate Histograms
  • Approximate Sets
  • Global Counters
  • Sink Routing
• Concepts
  • By Metric Type Behavior
  • Expiration
  • Other Notes
• Usage
• Setup
  • Clients
  • Einhorn Usage
  • Forwarding
    • Proxy
    • Static Configuration
    • Magic Tag
      • Global Counters And Gauges
      • Routing metrics
• Configuration
  • Configuration via Environment Variables
• Monitoring
  • At Local Node
    • Forwarding
  • At Global Node
  • Metrics
  • Error Handling
• Performance
  • Benchmarks
  • SO_REUSEPORT
  • TCP connections
  • TLS encryption and authentication
    • Performance implications of TLS
• Name

What Is Veneur?

Veneur (/vɛnˈʊr/, rhymes with “assure”) is a distributed, fault-tolerant pipeline for runtime data. It provides a server implementation of the DogStatsD protocol or SSF for aggregating metrics and sending them to downstream storage to one or more supported sinks. It can also act as a global aggregator for histograms, sets and counters.

More generically, Veneur is a convenient sink for various observability primitives with lots of outputs!

Use Case

Once you cross a threshold into dozens, hundreds or (gasp!) thousands of machines emitting metric data for an application, you've moved into that world where data about individual hosts is uninteresting except in aggregate form. Instead of paying to store tons of data points and then aggregating them later at read-time, Veneur can calculate global aggregates, like percentiles and forward those along to your time series database, etc.

Veneur is also a StatsD or DogStatsD protocol transport, forwarding the locally collected metrics over more reliable TCP implementations.

Here are some examples of why Stripe and other companies are using Veneur today:

• reducing cost by pre-aggregating metrics such as timers into percentiles
• creating a vendor-agnostic metric collection pipeline
• consolidating disparate observability data (from trace spans to metrics, and more!)
• improving efficiency over other metric aggregator implementations
• improving reliability by building a more resilient forwarding system over single points of failure

See Also

• A unified, standard format for observability primitives, the SSF
• A proxy for resilient distributed aggregation, veneur-proxy
• A command line tool for emitting metrics, veneur-emit
• A poller for scraping Prometheus metrics, veneur-prometheus
• The sinks supported by Veneur

We wanted percentiles, histograms and sets to be global. We wanted to unify our observability clients, be vendor agnostic and build automatic features like SLI measurement. Veneur helps us do all this and more!

Status

Veneur is currently handling all metrics for Stripe and is considered production ready. It is under active development and maintenance! Starting with v1.6, Veneur operates on a six-week release cycle, and all releases are tagged in git. If you'd like to contribute, see CONTRIBUTING!

Building Veneur requires Go 1.11 or later.

Features

Vendor And Backend Agnostic

Veneur has many sinks such that your data can be sent one or more vendors, TSDBs or tracing stores!

Modern Metrics Format (Or Others!)

Unify metrics, spans and logs via the Sensor Sensibility Format. Also works with DogStatsD, StatsD and Prometheus.

Global Aggregation

If configured to do so, Veneur can selectively aggregate global metrics to be cumulative across all instances that report to a central Veneur, allowing global percentile calculation, global counters or global sets.

For example, say you emit a timer foo.bar.call_duration_ms from 20 hosts that are configured to forward to a central Veneur. You'll see the following:

• Metrics that have been "globalized"
  • foo.bar.call_duration_ms.50percentile: the p50 across all hosts, by tag
  • foo.bar.call_duration_ms.90percentile: the p90 across all hosts, by tag
  • foo.bar.call_duration_ms.95percentile: the p95 across all hosts, by tag
  • foo.bar.call_duration_ms.99percentile: the p99 across all hosts, by tag
• Metrics that remain host-local
  • foo.bar.call_duration_ms.avg: by-host tagged average
  • foo.bar.call_duration_ms.count: by-host tagged count which (when summed) shows the total count of times this metric was emitted
  • foo.bar.call_duration_ms.max: by-host tagged maximum value
  • foo.bar.call_duration_ms.median: by-host tagged median value
  • foo.bar.call_duration_ms.min: by-host tagged minimum value
  • foo.bar.call_duration_ms.sum: by-host tagged sum value representing the total time

Clients can choose to override this behavior by including the tag veneurlocalonly.

Approximate Histograms

Because Veneur is built to handle lots and lots of data, it uses approximate histograms. We have our own implementation of Dunning's t-digest, which has bounded memory consumption and reduced error at extreme quantiles. Metrics are consistently routed to the same worker to distribute load and to be added to the same histogram.

Datadog's DogStatsD — and StatsD — uses an exact histogram which retains all samples and is reset every flush period. This means that there is a loss of precision when using Veneur, but the resulting percentile values are meant to be more representative of a global view.

Datadog Distributions

Because Veneur already handles "global" histograms, any DogStatsD packets received with type d — Datadog's distribution type — will be considered a histogram and therefore compatible with all sinks. Veneur does not send any metrics to Datadog typed as a Datadog-native distribution.

Approximate Sets

Veneur uses HyperLogLogs for approximate unique sets. These are a very efficient unique counter with fixed memory consumption.

Global Counters

Via an optional magic tag Veneur will forward counters to a global host for accumulation. This feature was primarily developed to control tag cardinality. Some counters are valuable but do not require per-host tagging.

Sink Routing

Veneur supports routing metrics to specific sinks using the metric_sink_routing configuration field with the structure:

metric_sink_routing:  # or
  - name: string
    match:  # or
      - name:
          kind: any | exact | prefix | regex
          value: string
        tags:  # and
          - kind: exact | prefix | regex
            unset: bool
            value: string
    sinks:
      matched:  # and
        - string
      not_matched:  # and
        - string

The metric_sink_routing field contains a list of routing rules, containing a name field for identifying the rule in logs, a list of matchers, and sinks. A matcher contains a name matcher for matching the name of a metric, and a list of tag matchers for matching tags the metric has; the name matcher and all of the tag matchers must match in order for the matcher to match a given metric.

The kind for the name matcher can be one of any, exact, prefix, or regex. The name matcher matches a metric name:

• any: always; the name of the metric is ignored, and the value field is unused.
• exact: if the metric name equals the value field.
• prefix: if the name starts with the value field.
• regex: if the name of the metric name matches the regex expression specified in the value field.

The kind of the tag matcher can be one of exact, prefix or regex. The tag matcher matches a metric tag:

• exact: if the tag equals the value field.
• prefix: if the tag starts with the value field.
• regex: if the tag matches the regex expression specified in the value field.

For a tag matcher to match a given metric, if the unset field is not set or is false, the tag matcher must match at least one tag in the metric; if the unset field is true, the tag matcher must match none of the tags in the metric.

If a metric matches any of the entries in the match field of a given rule, it is flushed to all of the sinks listed in the matched field; if a metric matches none of the matchers in a given rule, it is sent to all of the sinks listed in the not_matched section.

Concepts

• Global metrics are those that benefit from being aggregated for chunks — or all — of your infrastructure. These are histograms (including the percentiles generated by timers) and sets.
• Metrics that are sent to another Veneur instance for aggregation are said to be "forwarded". This terminology helps to decipher configuration and metric options below.
• Flushed, in Veneur, means metrics or spans processed by a sink.

By Metric Type Behavior

To clarify how each metric type behaves in Veneur, please use the following:

• Counters: Locally accrued, flushed to sinks (see magic tags for global version)
• Gauges: Locally accrued, flushed to sinks (see magic tags for global version)
• Histograms: Locally accrued, count, max and min flushed to sinks, percentiles forwarded to forward_address for global aggregation when set.
• Timers: Locally accrued, count, max and min flushed to sinks, percentiles forwarded to forward_address for global aggregation when set.
• Sets: Locally accrued, forwarded to forward_address for sinks aggregation when set.

Expiration

Veneur expires all metrics on each flush. If a metric is no longer being sent (or is sent sparsely) Veneur will not send it as zeros! This was chosen because the combination of the approximation's features and the additional hysteresis imposed by retaining these approximations over time was deemed more complex than desirable.

Other Notes

• Veneur aligns its flush timing with the local clock. For the default interval of 10s Veneur will generally emit metrics at 00, 10, 20, 30, … seconds after the minute.
• Veneur will delay it's first metric e