A framework for collecting and aggregating prometheus metrics
A framework for collecting and aggregating prometheus metrics
Prometheus Exporter allows you to aggregate custom metrics from multiple processes and export to Prometheus. It provides a very flexible framework for handling Prometheus metrics and can operate in a single and multiprocess mode.
To learn more see Instrumenting Rails with Prometheus (it has pretty pictures!)
Minimum Ruby of version 3.0.0 is required, Ruby 2.7 is EOL as of March 31st 2023.
There are some major changes in v1.x from v0.x.
Add this line to your application's Gemfile:
gem 'prometheus_exporter'And then execute:
$ bundleOr install it yourself as:
$ gem install prometheus_exporterSimplest way of consuming Prometheus exporter is in a single process mode.
…You can also choose custom quantiles for summaries and custom buckets for histograms.
summary = PrometheusExporter::Metric::Summary.new("load_time", "time to load page", quantiles: [0.99, 0.75, 0.5, 0.25])
histogram = PrometheusExporter::Metric::Histogram.new("api_time", "time to call api", buckets: [0.1, 0.5, 1])
In some cases (for example, unicorn or puma clusters) you may want to aggregate metrics across multiple processes.
Simplest way to achieve this is to use the built-in collector.
First, run an exporter on your desired port (we use the default bind to localhost and port of 9394):
$ prometheus_exporterAnd in your application:
require 'prometheus_exporter/client'
client = PrometheusExporter::Client.default
gauge = client.register(:gauge, "awesome", "amount of awesome")
gauge.observe(10)
gauge.observe(99, day: "friday")
Then you will get the metrics:
$ curl localhost:9394/metrics
# HELP collector_working Is the master process collector able to collect metrics
# TYPE collector_working gauge
collector_working 1
# HELP awesome amount of awesome
# TYPE awesome gauge
awesome{day="friday"} 99
awesome 10
Custom quantiles for summaries and buckets for histograms can also be passed in.
require 'prometheus_exporter/client'
client = PrometheusExporter::Client.default
histogram = client.register(:histogram, "api_time", "time to call api", buckets: [0.1, 0.5, 1])
histogram.observe(0.2, api: 'twitter')You can easily integrate into any Rack application.
In your Gemfile:
gem 'prometheus_exporter'In an initializer:
unless Rails.env.test?
require 'prometheus_exporter/middleware'
# This reports stats per request like HTTP status and timings
Rails.application.middleware.unshift PrometheusExporter::Middleware
endEnsure you run the exporter in a monitored background process:
$ bundle exec prometheus_exporterBy default, prometheus_exporter uses alias_method to instrument methods used by SQL and Redis as it is the fastest approach (see this article). You may desire to add additional instrumentation libraries beyond prometheus_exporter to your app. This can become problematic if these other libraries instead use prepend to instrument methods. To resolve this, you can tell the middleware to instrument using prepend by passing an instrument option like so:
Rails.application.middleware.unshift PrometheusExporter::Middleware, instrument: :prepend| Type | Name | Description |
|---|---|---|
| Counter | http_requests_total |
Total HTTP requests from web app |
| Summary | http_request_duration_seconds |
Time spent in HTTP reqs in seconds |
| Summary | http_request_redis_duration_seconds¹ |
Time spent in HTTP reqs in Redis, in seconds |
| Summary | http_request_sql_duration_seconds² |
Time spent in HTTP reqs in SQL in seconds |
| Summary | http_request_queue_duration_seconds³ |
Time spent queueing the request in load balancer in seconds |
| Summary | http_request_memcache_duration_seconds⁴ |
Time spent in HTTP reqs in Memcache in seconds |
All metrics have a controller and an action label.
http_requests_total additionally has a (HTTP response) status label.
To add your own labels to the default metrics, create a subclass of PrometheusExporter::Middleware, override custom_labels, and use it in your initializer.
…ruby
require 'prometheus_exporter/instrumentation'
if !PrometheusExporter::Instrumentation::Puma.started? PrometheusExporter::Instrumentation::Puma.start end
For Puma clustered mode
```ruby
# puma.rb config
after_worker_boot do
require 'prometheus_exporter/instrumentation'
# optional check, avoids spinning up and down threads per worker
if !PrometheusExporter::Instrumentation::Puma.started?
PrometheusExporter::Instrumentation::Puma.start
end
end
…
ruby
# e.g. config/initializers/resque.rb
require 'prometheus_exporter/instrumentation'
PrometheusExporter::Instrumentation::Resque.start
…
ruby
# e.g. config/initializers/good_job.rb
require 'prometheus_exporter/instrumentation'
PrometheusExporter::Instrumentation::GoodJob.start
…
bash
prometheus_exporter --unicorn-master /var/run/unicorn.pid --unicorn-listen-address 127.0.0.1:3000
# alternatively, if you're using unix sockets:
prometheus_exporter --unicorn-master /var/run/unicorn.pid --unicorn-listen-address /var/run/unicorn.sock
…
ruby
# person_collector.rb
class PersonCollector 21
@oldies.observe(1)
else
@youngies.observe(1)
end
end
def metrics
[@oldies, @youngies]
end
endShipping metrics then is done via:
PrometheusExporter::Client.default.send_json(type: "person", age: 40)To load the custom collector run:
$ bundle exec prometheus_exporter -a person_collector.rbCustom type collectors are the ideal place to collect global metrics, such as user/article counts and connection counts. The custom type collector runs in the collector, which usually runs in the prometheus exporter process.
Out-of-the-box we try to keep the prometheus exporter as lean as possible. We do not load all Rails dependencies, so you won't have access to your models. You can always ensure it is loaded in your custom type collector with:
unless defined? Rails
require File.expand_path("../../config/environment", __FILE__)
endThen you can collect the metrics you need on demand:
def metrics
user_count_gauge = PrometheusExporter::Metric::Gauge.new('user_count', 'number of users in the app')
user_count_gauge.observe User.count
[user_count_gauge]
end
…
ruby
class MyCustomCollector "app-server-01" }
counter = PrometheusExporter::Metric::Counter.new("web_requests", "number of web requests")
counter.observe(1, route: 'test/route')
counter.observeWill result in:
# HELP web_requests number of web requests
# TYPE web_requests counter
ruby_web_requests{hostname="app-server-01",route="test/route"} 1
ruby_web_requests{hostname="app-server-01"} 1When running the process for prometheus_exporter using bin/prometheus_exporter, there are several configurations that
can be passed in:
Usage: prometheus_exporter [options]
-p, --port INTEGER Port exporter should listen on (default: 9394)
-b, --bind STRING IP address exporter should listen on (default: localhost)
-t, --timeout INTEGER Timeout in seconds for metrics endpoint (default: 2)
--prefix METRIC_PREFIX Prefix to apply to all metrics (default: ruby_)
--label METRIC_LABEL Label to apply to all metrics (default: {})
-c, --collector FILE (optional) Custom collector to run
-a, --type-collector FILE (optional) Custom type collectors to run in main collector
-v, --verbose
-g, --histogram Use histogram instead of summary for aggregations
--auth FILE (optional) enable basic authentication using a htpasswd FILE
--realm REALM (optional) Use REALM for basic authentication (default: "Prometheus Exporter")
--unicorn-listen-address ADDRESS
(optional) Address where unicorn listens on (unix or TCP address)
--unicorn-master PID_FILE (optional) PID file of unicorn master process to monitor unicornThe following will run the process at
8080 (default 9394)0.0.0.0 (default localhost)1 second for metrics endpoint (default 2 seconds)foo_ (default ruby_){environment: "integration", foo: "bar"}prometheus_exporter -p 8080 \
-b 0.0.0.0 \
-t 1 \
--label '{"environment": "integration", "foo": "bar"}' \
--prefix 'foo_'You can use -b option to bind the prometheus_exporter web server to any IPv4 interface with -b 0.0.0.0,
any IPv6 interface with -b ::, or -b ANY to any IPv4/IPv6 interfaces available on your host system.
If you desire authentication on your /metrics route, you can enable basic authentication with the --auth option.
$ prometheus_exporter --auth my-htpasswd-fileAdditionally, the --realm option may be used to provide a customized realm for the challenge request.
Notes:
htpasswd formatted file before hand which contains one or more user:password entriescrypt eNo open issues yet, or sync has not completed.