Database storage grows unboundedly due to missing GC coverage in 4 core tables
Summary
We observed a consistent ~3% monthly growth in Concourse's PostgreSQL storage with no clear operational cause. After investigating the garbage collection subsystem we identified four tables that either have no collector at all, or have a collector that does not fully cover the deletion path.
| Table / Pattern | Issue | Effect |
|---|---|---|
builds |
Reaped build rows are never physically deleted — buildLogCollector marks reap_time but leaves the row in place. Only resource check builds are actually removed. |
Build rows accumulate forever for active pipelines, and drag build_resource_config_version_inputs/outputs and successful_build_outputs along with them via foreign keys |
team_build_events_{N} |
One-off builds write log events to per-team tables. buildLogCollector only iterates pipeline jobs, so one-off events are never reaped. |
Per-team event tables grow without bound until the entire team is deleted |
resource_config_versions |
No version cap exists for active resources. Versions are only removed via an explicit fly clear-versions call or when the resource itself is deleted. |
High-frequency check resources (e.g. a git resource on an active branch) accumulate the full version history indefinitely |
successful_build_outputs |
No GC collector exists for this table anywhere in the codebase. It is insert-only with no automatic delete path outside of full pipeline destruction. | Grows with every successful build and never shrinks for active pipelines |
Steps to Reproduce
- Run a Concourse installation with at least one active pipeline that has been running for several months.
- Connect to the backing PostgreSQL database.
- Run the following to check table sizes:
SELECT relname, pg_size_pretty(pg_total_relation_size(relid)), n_live_tup
FROM pg_stat_user_tables
WHERE relname IN (
'builds', 'successful_build_outputs',
'resource_config_versions',
'build_resource_config_version_inputs',
'build_resource_config_version_outputs'
)
ORDER BY pg_total_relation_size(relid) DESC;- Check how many build rows are reaped but never deleted:
SELECT COUNT(*) FROM builds WHERE reap_time IS NOT NULL;- Check successful_build_outputs row count against the number of reaped builds it references:
SELECT COUNT(*) FROM successful_build_outputs sbo
JOIN builds b ON b.id = sbo.build_id
WHERE b.reap_time IS NOT NULL;- Observe that the counts grow proportionally to pipeline activity and age, with no upper bound.
Expected Results
Completed build rows and their associated data should be physically removed after a configurable retention period. Resource version history should be capped per resource. The successful_build_outputs table should be pruned once a build's data is no longer needed for scheduling decisions
Actual Results
These tables grow monotonically for the lifetime of any active pipeline. The only way to reclaim space today is to destroy/recreate the pipeline entirely.
Web Node(s) configuration
Retention days and builds as well as global are configured properly on the instance
Worker(s) configuration
No response
Concourse Version
8.3.0
Browser (if applicable)
No response
Did this use to work?
not as far as I know
Source: concourse/concourse