[功能] 我们需要一个更强大的 Markdown 文档分块功能,以处理复杂的科学论文

Markdown structure segmentation has many defects, such as broken tables/formulas, lost title context, and misjudgment of sentence segmentation.

Desktop devices

• Operating system: Windows
• Browser: Google Chrome
• Easy Dataset version: latest version Using models
• Model provider: DeepSeek
• Model name: e.g. DeepSeek V4-pro
• Reproduction steps

1. Enter the document processing page.
2. Upload the markdown document and perform intelligent segmentation (select markdown structure segmentation).
3. View the segmentation in the intelligent segmentation box. You can see the following situations:

• Tables are arbitrarily split — visible page by page, no table detection, tables are cut by \n\n
• LaTeX formulas are corrupted — no $...$ / $$...$$ detection, sentence segmentation regular expression is incorrectly truncated at . in formulas
• Empty blocks may be generated — contaminating subsequent workflows
• Title context is lost — after long paragraphs are split, Part 2+ has no title, becoming an "orphan block"
• Accumulated logical defects — short paragraphs are "stuck", the end remains may be shorter than the set minimum segmentation size minLength
• Sentence segmentation is too simple — the regular expression /[^.!?。！？]+[.!?。！？]+/g is incorrectly truncated in decimal numbers, abbreviations, and code blocks Expected results For literature in various vertical fields, especially literature with complex structures and components (containing multiple special symbols, tables, various formula blocks, and even code blocks), it is possible to obtain the best segmentation results, and a multidimensional and reasonable scoring system can be used to evaluate the quality of each block, allowing quick navigation to the blocks with the worst, shortest, or longest segmentation results, making it convenient for self-optimization, and supporting:
• Custom editing/deleting block content
• Merger with adjacent blocks
• Self-splitting blocks
• Re-scoring and re-calculating
• Based on re-scoring and statistics, navigation to low-scoring, too long, or too short blocks for iterative optimization, ultimately making each block of appropriate length, with clear title context, complete semantics, and without broken tables, formulas, lists, etc., resulting in high-quality text blocks suitable for RAG or fine-tuning data set construction.