carbon-c-relay

carbon-c-relay(1) -- graphite relay, aggregator and rewriter

SYNOPSIS

carbon-c-relay -f config-file [ options ... ]

DESCRIPTION

carbon-c-relay accepts, cleanses, matches, rewrites, forwards and aggregates graphite metrics by listening for incoming connections and relaying the messages to other servers defined in its configuration. The core functionality is to route messages via flexible rules to the desired destinations.

carbon-c-relay is a simple program that reads its routing information from a file. The command line arguments allow to set the location for this file, as well as the amount of dispatchers (worker threads) to use for reading the data from incoming connections and passing them onto the right destination(s). The route file supports two main constructs: clusters and matches. The first define groups of hosts data metrics can be sent to, the latter define which metrics should be sent to which cluster. Aggregation rules are seen as matches. Rewrites are actions that directly affect the metric at the point in which they appear in the configuration.

For every metric received by the relay, cleansing is performed. The following changes are performed before any match, aggregate or rewrite rule sees the metric:

• double dot elimination (necessary for correctly functioning consistent hash routing)
• trailing/leading dot elimination
• whitespace normalisation (this mostly affects output of the relay to other targets: metric, value and timestamp will be separated by a single space only, ever)
• irregular char replacement with underscores (_), currently irregular is defined as not being in [0-9a-zA-Z-_:#], but can be overridden on the command line. Note that tags (when present and allowed) are not processed this way.

OPTIONS

These options control the behaviour of carbon-c-relay.

• -v: Print version string and exit.

• -d: Enable debug mode, this prints statistics to stdout and prints extra messages about some situations encountered by the relay that normally would be too verbose to be enabled. When combined with -t (test mode) this also prints stub routes and consistent-hash ring contents.

• -s: Enable submission mode. In this mode, internal statistics are not generated. Instead, queue pressure and metrics drops are reported on stdout. This mode is useful when used as submission relay which' job is just to forward to (a set of) main relays. Statistics about the submission relays in this case are not needed, and could easily cause a non-desired flood of metrics e.g. when used on each and every host locally.

• -S: Enable iostat-like mode where every second the current state of statistics are reported. This implies submission mode -s.

• -t: Test mode. This mode doesn't do any routing at all, but instead reads input from stdin and prints what actions would be taken given the loaded configuration. This mode is very useful for testing relay routes for regular expression syntax etc. It also allows to give insight on how routing is applied in complex configurations, for it shows rewrites and aggregates taking place as well. When -t is repeated, the relay will only test the configuration for validity and exit immediately afterwards. Any standard output is suppressed in this mode, making it ideal for start-scripts to test a (new) configuration.

• -f config-file: Read configuration from config-file. A configuration consists of clusters and routes. See CONFIGURATION SYNTAX for more information on the options and syntax of this file.

• -l log-file: Use log-file for writing messages. Without this option, the relay writes both to stdout and stderr. When logging to file, all messages are prefixed with MSG when they were sent to stdout, and ERR when they were sent to stderr.

• -p port: Listen for connections on port port. The port number is used for both TCP, UDP and UNIX sockets. In the latter case, the socket file contains the port number. The port defaults to 2003, which is also used by the original carbon-cache.py. Note that this only applies to the defaults, when listen directives are in the config, this setting is ignored.

• -w workers: Use workers number of threads. The default number of workers is equal to the amount of detected CPU cores. It makes sense to reduce this number on many-core machines, or when the traffic is low.

• -b batchsize: Set the amount of metrics that sent to remote servers at once to batchsize. When the relay sends metrics to servers, it will retrieve batchsize metrics from the pending queue of metrics waiting for that server and send those one by one. The size of the batch will have minimal impact on sending performance, but it controls the amount of lock-contention on the queue. The default is 2500.

• -q queuesize: Each server from the configuration where the relay will send metrics to, has a queue associated with it. This queue allows for disruptions and bursts to be handled. The size of this queue will be set to queuesize which allows for that amount of metrics to be stored in the queue before it overflows, and the relay starts dropping metrics. The larger the queue, more metrics can be absorbed, but also more memory will be used by the relay. The default queue size is 25000.

• -L stalls: Sets the max mount of stalls to stalls before the relay starts dropping metrics for a server. When a queue fills up, the relay uses a mechanism called stalling to signal the client (writing to the relay) of this event. In particular when the client sends a large amount of metrics in very short time (burst), stalling can help to avoid dropping metrics, since the client just needs to slow down for a bit, which in many cases is possible (e.g. when catting a file with nc(1)). However, this behaviour can also obstruct, artificially stalling writers which cannot stop that easily. For this the stalls can be set from 0 to 15, where each stall can take around 1 second on the client. The default value is set to 4, which is aimed at the occasional disruption scenario and max effort to not loose metrics with moderate slowing down of clients.

• -C CAcertpath: Read CA certs (for use with TLS/SSL connections) from given path or file. When not given, the default locations are used. Strict verfication of the peer is performed, so when using self-signed certificates, be sure to include the CA cert in the default location, or provide the path to the cert using this option.

• -T timeout: Specifies the IO timeout in milliseconds used for server connections. The default is 600 milliseconds, but may need increasing when WAN links are used for target servers. A relatively low value for connection timeout allows the relay to quickly establish a server is unreachable, and as such failover strategies to kick in before the queue runs high.

• -c chars: Defines the characters that are next to [A-Za-z0-9] allowed in metrics to chars. Any character not in this list, is replaced by the relay with _ (underscore). The default list of allowed characters is -_:#.

• -m length: Limits the metric names to be of at most length bytes long. Any lines containing metric names larger than this will be discarded.

• -M length Limits the input to lines of at most length bytes. Any excess lines will be discarded. Note that -m needs to be smaller than this value.

• -H hostname: Override hostname determined by a call to gethostname(3) with hostname. The hostname is used mainly in the statistics metrics carbon.relays.. sent by the relay.

• -B backlog: Sets TCP connection listen backlog to backlog connections. The default value is 32 but on servers which receive many concurrent connections, this setting likely needs to be increased to avoid connection refused errors on the clients.

• -U bufsize: Sets the socket send/receive buffer sizes in bytes, for both TCP and UDP scenarios. When unset, the OS default is used. The maximum is also determined by the OS. The sizes are set using setsockopt with the flags SO_RCVBUF and SO_SNDBUF. Setting this size may be necessary for large volume scenarios, for which also -B might apply. Checking the Recv-Q and the receive errors values from netstat gives a good hint about buffer usage.

• -E: Disable disconnecting idle incoming connections. By default the relay disconnects idle client connections after 10 minutes. It does this to prevent resources clogging up when a faulty or malicious client keeps on opening connections without closing them. It typically prevents running out of file descriptors. For some scenarios, however, it is not desirable for idle connections to be disconnected, hence passing this flag will disable this behaviour.

• -D: Deamonise into the background after startup. This option requires -l and -P flags to be set as well.

• -P pidfile: Write the pid of the relay process to a file called pidfile. This is in particular useful when daemonised in combination with init managers.

• -O threshold: The minimum number of rules to find before trying to optimise the ruleset. The default is 50, to disable the optimiser, use -1, to always run the optimiser use 0. The optimiser tries to group rules to avoid spending excessive time on matching expressions.

CONFIGURATION SYNTAX

The config file supports the following syntax, where comments start with a # character and can appear at any position on a line and suppress input until the end of that line:

…

CLUSTERS

Multiple clusters can be defined, and need not to be referenced by a match rule. All clusters point to one or more hosts, except the file cluster which writes to files in the local filesystem. host may be an IPv4 or IPv6 address, or a hostname. Since host is followed by an optional : and port, for IPv6 addresses not to be interpreted wrongly, either a port must be given, or the IPv6 address surrounded by brackets, e.g. [::1]. Optional transport and proto clauses can be used to wrap the connection in a compression or encryption layer or specify the use of UDP or TCP to connect to the remote server. When omitted the connection defaults to a plain TCP connection. type can only be linemode at the moment, e.g. Python's pickle mode is not supported.

DNS hostnames are resolved to a single address, according to the preference rules in RFC 3484. The any_of, failover and forward clusters have an explicit useall flag that enables expansion for hostnames resolving to multiple addresses. Using this option, each address of any type becomes a cluster destination. This means for instance that both IPv4 and IPv6 addresses are added.

There are two groups of cluster types, simple forwarding clusters and consistent hashing clusters.

• forward and file clusters

  The forward and file clusters simply send everything they receive to the defined members (host addresses or files). When a cluster has multiple members, all incoming metrics are sent to all members, basically duplicating the input metric stream over all members.

• any_of cluster

  The any_of cluster is a small variant of the forward cluster, b