Deterministic bitpacking encoder crash during CHECKPOINT — reproduces across 1.5.2 and 1.6.0-dev12

What happens?

A specific data pattern in a single table causes the bitpacking encoder to produce an out-of-range offset during CHECKPOINT, crashing the database with FATAL Error: Bitpacking offset is out of range at block "3696" - corrupt database file. Two independently built database files, with different hashes and sizes, reproduce the same bitpacking error at block 3696 under the same workload. This makes a deterministic data-pattern/storage-path bug more likely than a one-off filesystem corruption.

Tested on DuckDB 1.5.2 and 1.6.0-dev12 (nightly) — both reproduce.

Environment

• OS: macOS 15.6, Apple M4
• Python: 3.11.9 (python.org build)
• DuckDB versions tested: 1.5.2 (production), 1.6.0-dev12 (nightly, installed ~2026-05-04)
• Connection method: Python duckdb package

Error message (verbatim from logs)

FATAL Error: Failed to create checkpoint because of error: FATAL Error: Failed to create checkpoint: INTERNAL Error: Bitpacking offset is out of range at block "3696" - corrupt database file

After this error, the database enters an unrecoverable state for the remainder of the process lifetime:

FATAL Error: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.

To Reproduce

The bug reproduces deterministically under normal write + scheduled CHECKPOINT workload against a specific database. We have two preserved corrupt database files available for private sharing:

Re-corruption sequence:

1. Restored database from a known-clean daily backup (pre-corruption, verified by MD5).
2. Started application with normal write workload (multi-table inserts across 44 tables in 4 schemas, WAL mode).
3. At T+5m 0.18s from boot, a scheduled CHECKPOINT fired and crashed with the identical bitpacking error at the same block 3696.

Both database files are physically different (different SHA256, different sizes, different modification times) yet produce the same error at the same block. This makes random disk error unlikely — the data pattern written to the affected table appears to deterministically trigger the encoder bug.

Minimal reproduction commands against the preserved database:

import duckdb

# This crashes with the bitpacking error:
con = duckdb.connect("sie_v3.duckdb")
con.execute("CHECKPOINT")

# This succeeds — read-only connections work:
con = duckdb.connect("sie_v3.duckdb", read_only=True)
con.execute("DESCRIBE raw.\"RAW__Vol_DVOL\"").fetchall()
con.close()

Affected table

The investigation isolated failures to scanning/checkpointing storage containing this table; all other 43 tables exported cleanly via per-table COPY TO in subprocess-isolated read-only connections.

CREATE TABLE raw."RAW__Vol_DVOL" (
    snap_date_utc DATE NOT NULL,
    snap_ts_utc VARCHAR NOT NULL,
    snap_ts_utc_ms BIGINT NOT NULL,
    dvol_level DOUBLE,
    symbol VARCHAR NOT NULL,
    exchange VARCHAR NOT NULL,
    ingest_ts_utc VARCHAR NOT NULL,
    run_id VARCHAR NOT NULL
);

8 columns, approximately 767 rows at time of corruption. A sibling table (12 columns, ~3,028 rows) written by the same application code path is unaffected.

Additional observations

• Read-only connections work. duckdb.connect(path, read_only=True) opens the corrupt database successfully. DESCRIBE on the affected table succeeds (metadata path, no data block scan). information_schema.tables queries succeed. Connection closes cleanly. Re-verified 18 days post-incident.

• Write-mode connection close triggers crash. Even without querying the affected table, closing a write-mode connection on a database containing the corrupt blocks triggers a crash — the connection teardown touches catalog/storage metadata that includes the affected table's internal state.

• EXPORT DATABASE blocked. The standard recovery tool scans all tables and crashes on the affected table's data blocks. Recovery required subprocess-isolated per-table Parquet export, skipping the affected table.

• DuckDB 1.4.x behavior. Under DuckDB 1.4.x (macOS, Apple M4), the same database causes a SIGBUS (signal 10) that kills the process before any error message is logged. This is observed via process exit code only, not log text. The 1.5.2+ error message is a significant diagnostic improvement over the 1.4.x behavior.

Workaround in production

Periodic/explicit CHECKPOINT disabled during degraded operation (wal_autocheckpoint set to 1TB, explicit checkpoint calls removed). Application runs in WAL-only mode. Final remediation was selective compaction: subprocess-isolated per-table Parquet export (skipping the affected table), reimport into a fresh database, and recreation of the affected table as empty from DDL. Stable under this configuration for 18+ days.

Corrupt database files

We have two preserved database files (original corruption + independent re-corruption from clean backup) available for private sharing if useful for diagnosis. Please advise on preferred sharing method.

Database statistics (from corrupt file):

• 44 tables across 4 schemas
• PRAGMA database_size: 3884 total blocks / 3882 used / 2 free
• Block 3696 (named in error) is within the used range

OS:

macOS 15.6, Apple M4, aarch64

DuckDB Version:

1.5.2 (also reproduces on 1.6.0-dev12 nightly)

DuckDB Client:

Python (duckdb package, Python 3.11.9)

Hardware:

Apple M4, 16 GB RAM

Full Name:

Zuzana Martinkova

Affiliation:

Independent

Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

• Yes, I have

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant data sets for reproducing the issue?

No - Other reason (please specify in the issue body)