Split up grype DB into per-provider archives

Today grype ships a single SQLite database containing vulnerability data from all vunnel providers (~20+ providers, ~4M+ records, ~1.6 GB uncompressed DB size, ~100 MB compressed archive size). This proposal explores splitting the database into one archive per vunnel provider (e.g. nvd.db, ghsa.db, ubuntu.db, redhat.db, etc.) and leveraging SQLite's ATTACH DATABASE to query them as a unified dataset at runtime.

This is not a design document... it's the start of a conversation about viability, trade-offs, and whether the benefits justify the complexity.

Motivations

Download size and selective data

The full DB archive is only getting larger. Not every user needs every provider's data on every run. Someone scanning an Alpine container has no use for the Debian, Ubuntu, or Red Hat datasets. Per-provider archives would allow users to download only what they need, significantly reducing bandwidth and disk usage for targeted use cases. In theory a component analysis could decide which providers to update (and which to leave as is).

Parallel downloads and migrations

A single monolithic archive is currently downloaded without parallelism. Per-provider archives could be downloaded in parallel, reducing wall-clock update time. The same applies to hydration (migration) — independent databases could be hydrated concurrently rather than iterating 4.5M records in a single pass.

Build and publish pipeline simplification

The current grype-db pipeline pulls data from all vunnel providers, aggregates it onto a single runner, then iterates the full dataset to build one DB. This is operationally expensive and creates a serialization bottleneck.

We already run vunnel providers in parallel during the data sync phase. Per-provider databases would extend that parallelism through the build and publish steps — each provider's workflow could independently pull, build, and publish its own DB archive without waiting for the others. A failure in one provider (e.g. an upstream API outage) would no longer block the entire pipeline.

Startup time

At 1.6 GB uncompressed, the current DB has meaningful impact on startup time, especially in CI environments or serverless contexts where the DB may need to be loaded fresh. Attaching only the relevant provider databases would reduce this overhead.

Independent release cadence

Different providers have different update frequencies and reliability characteristics. NVD data changes on a different schedule than GitHub Advisory data. Per-provider databases would allow each provider's data to be published on its own cadence without forcing a full rebuild of the entire DB.

Possible implementation paths

I don't want to dive too far into implementation details here, but I also want to show that there might be a good path for this if we think the above characteristics are a good enough idea to move forward. (I don't want to over-rotate on sqlite attach and am open to other options here).

Possible direction: SQLite ATTACH

SQLite natively supports attaching multiple database files to a single connection via ATTACH DATABASE. Queries can reference tables across attached databases using schema-qualified names (e.g. nvd.vulnerability_handles), and temporary views can unify them:

ATTACH 'nvd.db' AS nvd;
ATTACH 'ghsa.db' AS ghsa;

CREATE TEMP VIEW all_vulnerability_handles AS
SELECT *, 'nvd' AS source FROM nvd.vulnerability_handles
UNION ALL
SELECT *, 'ghsa' AS source FROM ghsa.vulnerability_handles;

This is lightweight — ATTACH is fast, temp views store no data, and SQLite's query planner pushes predicates down into each branch of the union. Indexes on individual databases remain effective.

The default limit is 10 attached databases, but SQLite can be compiled with up to 125 (SQLITE_MAX_ATTACHED). We have ~30+ providers today so this limit is relevant.

Foreign key IDs (e.g. blob_id) are scoped per-database, so cross-DB joins on auto-increment IDs are meaningless — but this aligns naturally with how grype already queries: you always resolve blobs within the same provider's dataset.

Challenges and open questions

Distribution and discovery

This is likely the hardest problem. Today, latest.json points to a single archive. With per-provider databases, how does a client discover which provider databases are available?

A naive approach is a manifest file listing all provider archives, but this reintroduces coupling — you need something to aggregate the list, and publishing a new provider DB requires updating the manifest. This partially defeats the goal of independent provider pipelines.

Alternatively, provider databases could follow a predictable URL convention (e.g. databases/v7/{provider}/latest.json), allowing clients to discover and fetch them independently. But then the client needs to know the set of providers, or discover them dynamically (which honestly could be a single providers.json or some such that would not change often).

There's likely a spectrum of solutions here with different coupling/complexity trade-offs.

Schema version consistency

All attached databases must share the same schema version. You cannot have nvd.db at schema v7.1 and ghsa.db at v7.2 — the client code expects a uniform schema across all attached databases. This means:

• Schema migrations need to be coordinated across all provider databases
• A schema bump requires rebuilding and republishing every provider's database
• The client needs to validate schema consistency across all attached databases before use

This tension between independent provider pipelines and coordinated schema versions needs careful thought.

Preserving the single-DB workflow

Users who build custom databases with grype-db (including Anchore's own infrastructure) must retain the ability to produce a single consolidated DB. The build command should continue to support this — ideally the per-provider split is an additive capability, not a forced migration. Whether the "single DB" mode becomes the composition of per-provider databases or remains its own distinct path is an open question.

Attach limit

SQLite's default SQLITE_MAX_ATTACHED is 10. We have 30+ providers. This means either:

• We need a custom SQLite build (or ensure our Go SQLite driver is compiled with a higher limit)
• We group providers into a smaller number of databases (e.g. by ecosystem or tier)
• We selectively attach only the databases relevant to a given scan

There are several ways this could go, and they're not mutually exclusive:

1. Full per-provider split: One database per vunnel provider, client attaches what it needs. Maximum flexibility, maximum complexity.

2. Tiered grouping: Group providers into a small number of databases by category (e.g. "linux-distros", "language-ecosystems", "nvd+epss+kev"). Fewer files, stays under attach limits, but less granular.

3. Client-driven selection: Ship per-provider databases but have the client determine which providers are relevant based on the SBOM being scanned (e.g. if there are no Debian packages, skip debian.db). This could dramatically reduce the effective dataset size per scan.

4. Hybrid: Ship per-provider databases for distribution and parallel builds, but have the client merge them into a single local DB on first use (like a local aggregation step). Simpler query path at the cost of a one-time merge.

Each of these has different trade-offs. None of these are proposals — they're meant to sketch the possibility space.

Performance characteristics

While ATTACH and temp views are lightweight, querying across 30+ attached databases via UNION ALL views has different performance characteristics than querying a single database with a single index. This needs benchmarking — the theory would work, but real-world performance with the actual data volumes and query patterns grype uses may surface surprises.