Dagster doesn't expose a endpoint out of the box.
The officially documented way to get Dagster metrics into Prometheus is the resource, which pushes metrics to a Pushgateway from inside a run.
That works, but it has a structural blind spot: the push only happens if code inside the run gets to call it.
A run that's OOM-killed, or crashes before reaching the push call, never reports anything.
The failure is silent from Prometheus's point of view.
A run sitting in because of a run-queue concurrency limit hasn't started user code yet, so it can't push either — you can't see queue backlog forming.
Prometheus's own docs are explicit that Pushgateway isn't meant to be a general substitute for pull-based scraping, only for short-lived batch jobs that genuinely can't be scraped.
So I built dagster-prometheus-exporter: a small standalone Go binary that polls Dagster's GraphQL API on an interval and derives metrics from whatever state Dagster itself already has — including runs that never got a chance to push anything.
Architecture The exporter keeps in-memory state, not an external store.
Scraping (writing state) and serving (reading state) are decoupled, so if a GraphQL call is slow or fails, still serves the last-known state instead of breaking.
Completed runs are fetched incrementally (watermark-based, not a full rescan every time), so the cost doesn't grow with total run history.
What it currently covers Metric What it answers How many runs are queued/starting/started, per job How long the oldest active run in a job has been stuck there — useful for spotting stalls / / Success/failure counts and timing for completed runs Queue backlog per tag — this is the one that needed the most digging, since in the GraphQL schema looks like the answer but is actually a separate op-pool concurrency mechanism and reports 0 regardless of run-queue backlog.
Ended up reading each queued run's own tags instead.
Whether a code location is currently failing to load (e.g. broken import) — independent of job-level metrics, since a broken location can't be inferred from run counts / Whether a schedule is running, and its last tick outcome / Same, for sensors Which version/commit is actually running — handy once you have more than one exporter pod Full label reference is in the README.
Trying it out brings up Dagster + the exporter + Prometheus + a pre-provisioned Grafana dashboard together.
For an existing Dagster deployment, there's a published image and a Helm chart: What's not covered yet Asset materialization status.
I've looked into the GraphQL shape for it ( + ) and the same kind of gotcha as the concurrency backlog shows up: only records successful materializations, so detecting a failed one means cross-referencing instead.
Tracked in #56 if anyone wants to compare notes.
Repo: https://github.com/HirofumiTsuda/dagster-prometheus-exporter — issues and PRs welcome.