upright

Upright

Upright is a self-hosted synthetic monitoring system. It provides a framework for running health check probes from multiple geographic sites and reporting metrics via Prometheus. Alerts can then be configured with AlertManager.

Site overview with world map
30-day uptime history









Probe status across all sites

Features

• Playwright Probes - Browser-based probes for user flows with video recording and logs
• HTTP Probes - Simple HTTP health checks with configurable expected status codes
• SMTP Probes - EHLO handshake verification for mail servers
• Traceroute Probes - Network path analysis with hop-by-hop latency tracking
• Multi-Site Support - Run probes from multiple geographic locations with staggered scheduling
• Observability - OTLP compatible, Prometheus metrics, OpenTelemetry tracing, and AlertManager support
• Configurable Authentication - OmniAuth integration with support for any OIDC provider

Not Included

• Notifications - Instead, Alertmanager is included for alerting and notifications
• Hosting - Instead, you can use a VPS from DigitalOcean, Hetzner, etc.

Components

• Rails engine
• SQLite
• Solid Queue for background and recurring jobs
• Mission Control - Jobs to monitor Solid Queue and manually enqueue probes
• Kamal for deployments
• Prometheus metrics for uptime queries and alerting
• AlertManager for notifications
• Open Telemetry Collector - logs, metrics and traces can be shipped to any OTLP compatible endpoint

Installation

[!NOTE] Upright is designed to be run in its own Rails app and deployed with Kamal.

Quick Start (New Project)

Create a new Rails application and install Upright:

rails new my-upright --database=sqlite3 --skip-test
cd my-upright
bundle add upright
bin/rails generate upright:install
bin/rails db:prepare

Start the server:

bin/dev

Visit http://app.my-upright.localhost:3000 to see your Upright instance.

Note: Upright uses subdomain-based routing. The app subdomain is the admin interface, while site-specific subdomains (e.g., nyc, lon) show probe results for each location. The .localhost TLD resolves to 127.0.0.1 on most systems.

What the Generator Creates

The upright:install generator creates:

• config/database.yml - Development and test split into primary (probe results), persistent (rollups and incidents, migrated from the gem) and queue (Solid Queue) databases
• config/recurring.yml - Probe schedules and the engine's housekeeping, health, rollup, incident and maintenance jobs
• config/initializers/upright.rb - Engine configuration
• config/initializers/content_security_policy.rb - Ready-to-enable CSP covering the engine's CDN dependencies
• config/sites.yml - Site definitions for each VPS you host Upright on
• config/prometheus/prometheus.yml - Prometheus configuration
• config/alertmanager/alertmanager.yml - AlertManager configuration
• config/otel_collector.yml - OpenTelemetry Collector configuration
• config/deploy.yml and Dockerfile - Kamal deployment, with Prometheus and Alertmanager pinned by image digest
• .kamal/secrets entries and .kamal/hooks/pre-deploy - The machine tokens and admin password as Kamal secrets, and a hook that refuses to deploy while a token is missing (see Machine tokens)
• probes/ - Directory for all HTTP, SMTP, Traceroute YAML config as well as Playwright probe classes

It also mounts the engine at / in your routes.

Configuration

Basic Setup

See config/initializers/upright.rb

Hostname Configuration

Upright uses subdomain-based routing. Configure your production hostname:

# config/initializers/upright.rb
Upright.configure do |config|
  config.hostname = "upright.com"
end

For local development, the hostname defaults to {service_name}.localhost (e.g., upright.localhost).

Site Configuration

Define your monitoring locations in config/sites.yml:

shared:
  sites:
    - code: nyc
      city: New York City
      country: US
      geohash: dr5reg
      provider: digitalocean

    - code: ams
      city: Amsterdam
      country: NL
      geohash: u17982
      provider: digitalocean
      stores_metrics: true
      primary: true

    - code: sfo
      city: San Francisco
      country: US
      geohash: 9q8yy
      provider: hetzner

Each site node identifies itself via the SITE_SUBDOMAIN environment variable, configured in your Kamal deploy.yml.

Two optional flags give a site a role beyond running probes: primary serves the app and status hostnames and runs the jobs writing the shared database, and stores_metrics runs a local Prometheus and Alertmanager. See Sites and their roles for what each one changes, the health metrics every site exports, and how daily rollups read across them.

Machine tokens

Two bearer tokens authenticate machine callers on the /prometheus and /alertmanager proxies. Each does one job:

Env var	Config	Who presents it	What it allows
PROMETHEUS_OTLP_TOKEN	config.otlp_token	The OpenTelemetry collector on every site	POST /prometheus/api/v1/otlp/v1/metrics, and nothing else
METRICS_READ_TOKEN	config.metrics_read_token	Peer sites computing rollups, and tooling	GET and HEAD on /prometheus and /alertmanager, except the /-/ lifecycle endpoints

Generate each with bin/rails secret. Set them as Kamal secrets, the same two values on every site, and nowhere in git. Outside development and test the app refuses to boot when either is missing or when both hold the same value, and .kamal/hooks/pre-deploy refuses to deploy for the same reasons before any container is replaced.

export PROMETHEUS_OTLP_TOKEN=$(bin/rails secret)
export METRICS_READ_TOKEN=$(bin/rails secret)
bin/kamal deploy

A token cannot reach any other route. The admin UI uses a session cookie instead.

Authentication

Static Credentials

Upright uses static credentials by default with username admin and the password taken from the ADMIN_PASSWORD environment variable.

[!IMPORTANT] There is no default password: ADMIN_PASSWORD must be set, and sign-in fails closed without it. Set it as a Kamal secret in production (the generated config/deploy.yml already lists it) and export it locally for development.

OpenID Connect

For production environments, Upright supports OpenID Connect (Logto, Keycloak, Duo, Okta, etc.):

# config/initializers/upright.rb
Upright.configure do |config|
  config.auth_provider = :openid_connect
  config.auth_options = {
    issuer: "https://your-tenant.logto.app/oidc",
    client_id: ENV["OIDC_CLIENT_ID"],
    client_secret: ENV["OIDC_CLIENT_SECRET"]
  }
end

Probe Result Cleanup

Upright automatically cleans up old probe results on a recurring schedule. You can configure the retention thresholds:

Upright.configure do |config|
  config.stale_success_threshold = 24.hours     # Delete successful results older than this (default: 24 hours)
  config.stale_failure_threshold = 30.days       # Delete failed results older than this (default: 30 days)
  config.failure_retention_limit = 20_000        # Keep at most this many failed results (default: 20,000)
end

Custom Probe Types

Upright ships with four built-in probe types: HTTP, Playwright, SMTP, and Traceroute. You can register your own to extend the system.

1. Register the type

Add it to your initializer so Upright knows about its name and icon:

# config/initializers/upright.rb
Upright.configure do |config|
  config.probe_types.register :ping, name: "Ping", icon: ""
end

2. Create the probe class

Add a Ruby class in your probes/ directory that extends FrozenRecord::Base and includes Upright::Probeable and Upright::ProbeYamlSource. Implement probe_type, probe_target, check, and on_check_recorded:

…

Secrets

config/deploy.yml reads RAILS_MASTER_KEY, PROMETHEUS_OTLP_TOKEN, METRICS_READ_TOKEN and ADMIN_PASSWORD from .kamal/secrets, which the generator points at environment variables of the same names. Kamal passes an empty value through when a variable is unset, so .kamal/hooks/pre-deploy checks the resolved secrets Kamal hands it and stops the deploy when either token is empty or both are the same. The app applies the same checks at boot. See Machine tokens.

Observability

Prometheus

Metrics are exposed via a Puma plugin at http://0.0.0.0:9394/metrics. Configure Prometheus to scrape:

scrape_configs:
  - job_name: upright
    static_configs:
      - targets: ['localhost:9394']

Metrics Exposed

• upright_probe_duration_seconds - Probe execution duration
• upright_probe_up - Probe status (1 = up, 0 = down)
• upright_http_response_status - HTTP response status code

Labels include: type, name, site_code, site_city, site_country

AlertManager

Example alert rules (prometheus/rules/upright.rules):

groups:
  - name: upright
    rules:
      - alert: ProbeDown
        expr: upright_probe_up == 0
        for: 5m
        labels:
          severity: "{{ $labels.alert_severity }}"
        annotations:
          summary: "Probe {{ $labels.name }} is down"

OpenTelemetry

Traces are automatically created for each probe execution. Configure your collector endpoint:

Upright.configure do |config|
  config.otel_endpoint = "https://otel.example.com:4318"
end

Local Development

Setup

bin/setup

This installs dependencies, prepares the database, and starts the dev server.

Running Services

Start supporting Docker services (Playwright server, etc.):

bin/services

Running the Server

bin/dev

Visit http://app.upright.localhost:3000 and sign in with:

• Username: admin
• Password: the value of the ADMIN_PASSWORD env var, e.g. ADMIN_PASSWORD=upright bin/dev (a throwaway value like this is fine for local testing only — never for a deployed site)

Testing Playwright Probes

Run probes with a visible browser window:

HEADLESS=false bin/rails console

Probes::Playwright::MyServiceAuthProbe.check

Upgrading Playwright

Playwright versions are pinned via Upright::PLAYWRIGHT_VERSION in lib/upright/version.rb. This drives the Ruby gem and npm package versions. To upgrade:

1. Update PLAYWRIGHT_VERSION in lib/upright/version.rb
2. Update the version in package.json
3. Run bin/setup (or manually: npm install && npx playwright install chromium)
4. Run bin/rails test to verify compatibility
5. Commit the updated package.json and package-lock.json

Viewing Playwright Traces

Upright stores a Playwright trace as a probe result artifact and does not serve the Playwright Trace Viewer. The viewer renders a trace's tags and attributes as HTML in its own origin, so serving it from Upright's hostname would let a trace run script against an admin session.

A trace artifact links to config.trace_viewer_url, which defaults to upstream's hosted viewer at https://trace.playwright.dev. That viewer runs entirely in the browser, and being on its own registrable domain is what keeps a trace's contents off Upright's origin.

Following the link hands that origin a URL it can read for 24 hours. Set config.trace_viewer_url to a viewer you host to keep traces to yourself, or assign nil to keep them download-only:

npx playwright show-trace trace.zip

A URL under config.hostname raises Upright::ConfigurationError, since that would put the trace back on the admin origin.

That trace URL is served by Upright::TracesController, not Active Storage. The viewer fetches it from its own or