Etsy 的 statsd 在 Go 中的实现,支持标签
An implementation of [Etsy's][etsy] [statsd][statsd] in Go, based on original code from @kisielk.
The project provides both a server called "gostatsd" which works much like Etsy's version, but also provides a library for developing customized servers.
Backends are pluggable and only need to support the backend interface.
Being written in Go, it is able to use all cores which makes it easy to scale up the server based on load.
[](# Go version needs to be the same in: CI config, README, Dockerfiles, and Makefile) Gostatsd currently targets Go 1.26.7. If you are compiling from source, please ensure you are running this version.
From the gostatsd directory run make build. The binary will be built in build/bin//gostatsd.
You will need to install the Golang build dependencies by running make setup in the gostatsd directory. This must be done before the first build,
and again if the dependencies change. A protobuf installation is expected to be found in the tools/
directory. Managing this in a platform agnostic way is difficult, but PRs are welcome. Hopefully it will be sufficient to use the generated protobuf
files in the majority of cases.
If you are unable to build gostatsd please check your Go version, and try running make setup again before reporting a bug.
gostatsd --help gives a complete description of available options and their
defaults. You can use make run to run the server with just the stdout backend
to display info on screen.
You can also run through docker by running make run-docker which will use docker-compose
to run gostatsd with a graphite backend and a grafana dashboard.
While not generally tested on Windows, it should work. Maximum throughput is likely to be better on a linux system, however.
The server listens for UDP packets by default. You can use unix sockets providing an absolute path to the socket
in the metrics-addr configuration option. The socket mode used in this case is SOCK_DGRAM.
Note that using unix sockets will only work on linux and that it will ignore conn-per-reader configuration option.
The server can currently run in two modes: standalone and forwarder. It is configured through the top level
server-mode configuration setting. The default is standalone.
In standalone mode, raw metrics are processed and aggregated as normal, and aggregated data is submitted to
configured backends (see below)
This configuration mode allows the following configuration options:
expiry-interval: interval before metrics are expired, see Metric expiry and persistence section. Defaults
to 5m. 0 to disable, -1 for immediate.expiry-interval-counter: interval before counters are expired, defaults to the value of expiry-interval.expiry-interval-gauge: interval before gauges are expired, defaults to the value of expiry-interval.expiry-interval-set: interval before sets are expired, defaults to the value of expiry-interval.expiry-interval-timer: interval before timers are expired, defaults to the value of expiry-interval.flush-aligned: whether or not the flush should be aligned. Setting this will flush at an exact time interval. With
a 10 second flush-interval, if the service happens to be started at 12:47:13, then flushing will occur at 12:47:20,
12:47:30, etc, rather than 12:47:23, 12:47:33, etc. This removes query time ambiguity in a multi-server environment.
Defaults to false.flush-interval: duration for how long to batch metrics before flushing. Should be an order of magnitude less than
the upstream flush interval. Defaults to 1s.flush-offset: offset for flush interval when flush alignment is enabled. For example, with an offset of 7s and an
interval of 10s, it will flush at 12:47:10+7 = 12:47:17, etc.ignore-host: indicates whether or not an explicit host field will be added to all incoming metrics and events.
Defaults to falsemax-readers: the number of UDP receivers to run. Defaults to 8 or the number of logical cores, whichever is less.max-parsers: the number of workers available to parse metrics. Defaults to the number of logical cores.max-workers: the number of aggregators to process metrics. Defaults to the number of logical cores.max-queue-size: the size of the buffers between parsers and workers. Defaults to 10000, monitored via
channel.* metric, with dispatch_aggregator_batch and dispatch_aggregator_map channels.max-concurrent-events: the maximum number of concurrent events to be dispatching. Defaults to 1024, monitored
via channel.* metric, with backend_events_sem channel.estimated-tags: provides a hint to the system as to how many tags are expected to be seen on any particular metric,
so that memory can be pre-allocated and reducing churn. Defaults to 4. Note: this is only a hint, and it is safe
to send more.log-raw-metric: logs raw metrics received from the network. Defaults to false.metrics-addr: the address to listen to metrics on. Defaults to :8125. Using a file path instead of host:port
will create a Unix Domain Socket in the specified path instead of using UDP.namespace: a namespace to prefix all metrics with. Defaults to ''.statser-type: configures where internal metrics are sent to. May be internal which sends them to the internal
processing pipeline, logging which logs them, null which drops them. Defaults to internal, or null if the
NewRelic backend is enabled.percent-threshold: configures the "percentiles" sent on timers. Space separated string. Defaults to 90.heartbeat-enabled: emits a metric named heartbeat every flush interval, tagged by version and commit.
Defaults to false.receive-batch-size: the number of datagrams to attempt to read. It is more CPU efficient to read multiple, however
it takes extra memory. See [Memory allocation for read buffers] section below for details. Defaults to 50.receive-buffer-size: the size of single buffer that's used to store datagram during read. Number of buffers is defined
by receive-batch-size. Default to 65535 what is theoretical maximum UDP packet size.conn-per-reader: attempts to create a connection for every UDP receiver. Not supported by all OS versions. It will
be ignored when unix sockets are used for the connection.
Defaults to false.bad-lines-per-minute: the number of metrics which fail to parse to log per minute. This is used to prevent a bad
client spamming malformed statsd data, while still logging some information to enable troubleshooting. Defaults to 0.hostname: sets the hostname on internal metricstimer-histogram-limit: specifies the maximum number of buckets on histograms. See [Timer histograms] below.In forwarder mode, raw metrics are collected from a frontend, and instead of being aggregated they are sent via http
to another gostatsd server after passing through the processing pipeline (cloud provider, static tags, filtering, etc).
A forwarder server is intended to run on-host and collect metrics, forwarding them on to a central aggregation
service. At present the central aggregation service can only scale vertically, but horizontal scaling through
clustering is planned.
Aligned flushing is deliberately not supported in forwarder mode, as it would impact the central aggregation server
due to all for forwarder nodes transmitting at once, and the expectation that many forwarding flushes will occur per
central flush anyway.
Configuring forwarder mode requires a configuration file, with a section named http-transport. The raw version
spoken is not configurable per server (see HTTP.md for version guarantees). The configuration section allows the
following configuration options:
compress: boolean indicating if the payload should be compressed. Defaults to truecompression-type: compression algorithm to use, one of zlib or lz4. Defaults to zlib. Note lz4 is non-standard
so make sure the downstream consumer can handle Content-Encoding='lz4'.compression-level: compression level to use (0-9). 0 = best speed, 9 = best compression. Defaults to 9.api-endpoint: configures the endpoint to submit raw metrics to. This setting should be just a base URL, for example
https://statsd-aggregator.private, with no path. Required, no defaultmax-requests: maximum number of requests in flight. Defaults to 1000 (which is probably too high)concurrent-merge: maximum number of concurrent goroutines allowed to merge metrics before forwarding. Defaults to 1 for backward-compatibilitymax-request-elapsed-time: duration for the maximum amount of time to try submitting data before giving up. This
includes retries. Defaults to 30s (which is probably too high). Setting this value to -1 will disable retries.consolidator-slots: number of slots in the metric consolidator. Memory usage is a function of this. Lower values
may cause blocking in the pipeline (back pressure). A UDP only receiver will never use more than the number of
configured parsers (--max-parsers option). Defaults to the value of --max-parsers, but may require tuning for
HTTP based servers.transport: see TRANSPORT.md for how to configure the transport.custom-headers : a map of strings that are added to each request sent to allow for additional network routing / request inspection.
Not required, default is empty. Example: --custom-headers='{"region" : "us-east-1", "service" : "event-producer"}'dynamic-headers : similar with custom-headers, but the header values are extracted from metric tags matching the
provided list of string. Tag names are canonicalized by replacing underscores with hyphens. If a tag is specified
in both custom-header and dynamic-header, the vaule set by custom-header takes precedence. Not required, default
is empty. Example: --dynamic-headers='["region", "service"]'.
This is an experimental feature and it may be removed or changed in future versions.fail-fast-on-non-retryable-errors: boolean flag that controls whether to immediately drop messages that encounter
non-retryable HTTP errors (e.g., 4xx status codes indicating client errors). When set to true, non-retryable errors
immediately drop the message without retry attempts. When set to false, even non-retryable errors will attempt retries
via exponential backoff up to the max-request-elapsed-time limit. Note: Some errors (e.g., TLS certificate validation
failures) are always treated as non-retryable. Defaults to false.The following settings from the previous section are also supported:
expiry-*ignore-hostmax-readersmax-parsersestimated-tagslog-raw-metricmetrics-addrnamespacestatser-typeheartbeat-enabledreceive-batch-sizeconn-per-readerbad-lines-per-minutehostnamelog-raw-metricGostatsd can be run as a lambda extension in forwarder mode. The metrics are flushed at the end of each lambda invocation by default. The flush interval is ignored for your custom metrics, internal metrics are still flushed on a best effort basis using the configured flush interval.
To support flushes based on the runtime function, a lambda telemetry server is started at the reserved lambda hostname
sandbox on port 8083. This can be configured by setting the lambda-extension-telemetry-address configuration parameter.
This will need to be done if port 8083 is not available within the lambda runtime.
暂无开放 Issues,或尚未同步最近议题。