#3079·tantivy

Are orphaned segment files recoverable after writer process crash?

Author: Syed-Shahidh-IlhanCreated Sep 7, 2026Updated Sep 9, 2026

We're running Tantivy as the index writer in a long-running service where writer processes occasionally get SIGKILL'd (OOM). After a crash, the index directory has segment component files on disk, but meta.json still shows "segments": [], "opstamp": 0 — the writer was instantiated and accepted documents, but commit() was never called (or never completed).

We sampled ~800 such directories from a workload. The on-disk state is consistent:

$ ls -la /data/indices/.../2026-09-08_20-47/
total 16
-rw-------  258  .managed.json
-rw-r--r--    0  .tantivy-writer.lock
-rw-r--r--    0  7a8c2a81201346f686c5126823dc31d3.fast
-rw-r--r--    0  7a8c2a81201346f686c5126823dc31d3.fieldnorm
-rw-r--r--    0  7a8c2a81201346f686c5126823dc31d3.idx
-rw-r--r--    0  7a8c2a81201346f686c5126823dc31d3.pos
-rw-r--r--    0  7a8c2a81201346f686c5126823dc31d3.store
-rw-r--r--    0  7a8c2a81201346f686c5126823dc31d3.term
-rw-------  4014 meta.json
json
// meta.json (truncated)
{
  "segments": [],
  "opstamp": 0,
  "schema": [ ... ]
}

Of ~800 directories, ~799 have all six component files at 0 bytes (writer created, no data flushed to disk). One had a 123-byte .store (partial docstore spill, no other component files populated):

-rw-rw-r--    0  43a2e3659e12...fast
-rw-rw-r--    0  43a2e3659e12...fieldnorm
-rw-rw-r--    0  43a2e3659e12...idx
-rw-rw-r--    0  43a2e3659e12...pos
-rw-rw-r--  123  43a2e3659e12...store    ← partial spill
-rw-rw-r--    0  43a2e3659e12...term

Still segments: [], opstamp: 0.

Three questions:

  1. Is this the expected on-disk state after a writer crash before commit()? My read: Index::create_in_dir() writes the initial meta.json and .managed.json; index.writer() acquires the lock and allocates a segment UUID with placeholder files; documents are buffered in memory until commit() calls prepare_commit()finalize() which serializes the component files with footers. If the process dies before commit(), the component files stay at 0 bytes (or partially spilled) and meta.json is never updated. Is that correct?

  2. On garbage_collect_files: reading the code, it looks like GC runs inside prepare_commit() and diffs files on disk against files referenced by segments in the SegmentManager (loaded from meta.json). With segments: [], the living set is empty, so GC would delete the orphaned component files on the next commit. But in our case, nothing ever re-opens these directories to run another commit — the process restarts with a new writer and new directory. Is there a recommended way to trigger GC against an existing directory, or is the intended cleanup just rm -rf of the whole directory?

  3. Is there any path to recovering data from the partially-spilled case (123-byte .store, everything else 0)? My assumption is no — without commit() having run, the component files have no footer, the segment isn't registered, and the in-memory state that would have completed the write is gone with the process. Just want to confirm there's no truncation-tolerant reader path I'm missing.

We're planning to sweep these directories at startup. Just want to confirm our understanding before doing so.