[Bug]: Bundled ParadeDB BM25 index corruption breaks document indexing and hybrid search

Affected Component

Vector Database

Bug Description

The PostgreSQL/ParadeDB BM25 index used by the bundled PostgreSQL deployment can enter an internally inconsistent state. Once this happens, both document indexing and hybrid search fail.

The application reports a generic vector-store write failure (VECTORSTORE_WRITE_FAILED: batch index failed), while the underlying PostgreSQL error is:

ERROR: assertion failed: item_pointer_is_valid(ctid) (SQLSTATE XX000)

In the affected deployment, rebuilding public.embeddings_search_idx immediately restored keyword retrieval and hybrid search. Reprocessing the failed document then completed vector insertion successfully.

Environment details:

• App Version: v0.7.0
• UI Version: v0.7.0
• PostgreSQL image: paradedb/paradedb:v0.22.2-pg17
• pg_search extension: 0.22.2
• Keyword index: public.embeddings_search_idx (USING bm25)

Steps to Reproduce

1. Deploy WeKnora v0.7.0 with the bundled ParadeDB PostgreSQL image.

2. Import and process a large document. In our case, the failure occurred with 10,553 chunks.

3. Run document processing or hybrid search using PostgreSQL keyword retrieval.

4. Observe VECTORSTORE_WRITE_FAILED / HTTP 500 responses and the PostgreSQL assertion failure shown below.

5. Run:

   sql复制

   REINDEX INDEX public.embeddings_search_idx;

6. Repeat the same hybrid search and document reprocessing. Both succeed again.

The exact trigger that invalidates the BM25 index is not yet known, but the failure and recovery are deterministic once the bad index state exists.

Expected Behavior

The bundled BM25 index should remain valid during normal document ingestion, deletion, and reprocessing. If an invalid index state is detected, WeKnora should surface a diagnostic error and provide or perform a safe recovery instead of returning generic indexing and search failures indefinitely.

Actual Behavior

• Document processing fails during the embedding/vector-store write stage with VECTORSTORE_WRITE_FAILED: batch index failed.
• Hybrid search returns HTTP 500.
• PostgreSQL reports assertion failed: item_pointer_is_valid(ctid).
• The problem persists until the BM25 index is rebuilt manually.
• REINDEX INDEX public.embeddings_search_idx restores keyword retrieval, hybrid search, and document reprocessing.

WeKnora Version

App Version: v0.7.0, UI Version: v0.7.0

Deployment Method

Docker Compose

Operating System

Ubuntu 24.04.4 LTS, Linux kernel 6.17.0-35-generic

Relevant Logs

VECTORSTORE_WRITE_FAILED
batch index failed

ERROR: assertion failed: item_pointer_is_valid(ctid) (SQLSTATE XX000)

# Successful temporary recovery
REINDEX INDEX public.embeddings_search_idx;
REINDEX

Confirmation

• I have searched existing issues and confirmed this is a new one