#955·qmd

Bug report: chunker splits markdown tables mid-rows

Author: yonikremerCreated Sep 14, 2026Updated Sep 14, 2026

Summary

The smart chunker (src/store.ts) never splits inside fenced code blocks but freely splits markdown tables between rows. Any table larger than the chunk budget (~900 tokens / 3600 chars) is cut into row fragments, stranding cells from their header row. Tables are the exact content (spec parameter tables, converted .docx) where a split hurts most.

Repro

  1. Build a markdown doc with one table larger than CHUNK_SIZE_CHARS (3600), e.g. 60+ rows of | parameter | type | default | description |.
  2. Call chunkDocument(content) (src/store.ts) or index + embed the file.
  3. The 900-token target boundary lands mid-table; output chunks break between table rows. The second chunk's rows have no header — "42 | ms | timeout" with no column names.

Root cause (with refs, all src/store.ts @04e4dbd)

  • BREAK_PATTERNS (~line 172) scores headings, codeblock boundaries, hr, blank lines, list items, and bare newlines — there is no table-row/table-boundary pattern, and every "\n" (including each table row start) is a score-1 breakpoint via the /\n/g fallback.
  • findCodeFences (~line 230) is the only region protection, and findBestCutoff (~line 270) only skips breakpoints inside code-fence regions. Table interiors are unprotected, so a score-1 row boundary inside a table is a legal cut.
  • chunkDocumentWithBreakPoints (~line 366) then cuts there whenever the target lands mid-table. Confirmed: grep for "table" in src/store.ts returns only SQLite/FTS5 hits; no test in test/ covers table chunking.

Impact

  • Converted .docx / spec docs are table-dense; parameter tables routinely exceed one chunk.
  • Split rows lose headers: cells without column names are unanswerable ("42" = default? max? timeout?) and embed poorly (row text without header context drifts in vector space).
  • Downstream dedupe/entity passes see headerless fragments as distinct low-quality claims.

Proposed fix (mirrors the existing fence logic)

  1. Add findTableRegions(content): contiguous runs of lines starting with "|" (after optional whitespace), requiring a delimiter row (|:?-+:?|) to qualify, mirroring findCodeFences including an "unclosed runs to end of doc" rule. Fence detection takes precedence (a "|" run inside a code fence is code, not a table).
  2. In findBestCutoff, skip breakpoints strictly inside a table region (same as fences).
  3. Score the table start boundary as a decent break (suggest ~60, like hr) and the table end likewise, so chunks prefer to end before / start after a table rather than stopping nearby.
  4. Oversized single tables (> maxChars): keep whole if within ~1.5x budget (a whole table is worth an oversized chunk); beyond that, fall back to row-boundary cuts but prepend the header row to each continuation chunk.
  5. Edge cases: "|" in prose or inline code spans (require line-start "|" + delimiter row); tables inside blockquotes/lists (">" / "-" prefix before "|"); single "|" lines are not tables.