[功能] 功能建议:支持 Obsidian 知识库作为生成数据集时的上下文参考层

作者: virologist创建于 2026年4月11日更新于 2026年5月1日
标签enhancement

Background and pain points The existing workflow is linear: upload documents → slice → generate QA. This process is fine for covering a single document, but in practice, there is a structural flaw — when LLM generates each set of QA, it can only see the current slice, and has no idea: - where this knowledge point is located in the entire domain system - what relationships exist between this concept and other concepts (e.g. the inference chain between "chicken infectious fowlpox" and "vaccine design") - which questions have been covered in other slices, and which areas are still unaddressed. As a result, the generated questions are often superficial and repetitive, lacking cross-concept inference, which is the core weakness in improving the quality of the fine-tuning dataset. Why choose the Obsidian knowledge base Many users who do domain-specific fine-tuning with Easy Dataset (especially small teams) already maintain their domain knowledge notes in Obsidian. These notes are not original documents, but structured knowledge graphs that have been personally digested and organized, containing: - [[bi-directional]] relationships between concepts - a framework for the user's understanding of the domain and inference paths - a hierarchical relationship between original materials and distilled conclusions. If the relevant knowledge can be referenced when generating QA, the generated questions will be more in-depth, the answer inference chain will be more complete, and the quality of the fine-tuning of the dataset will be significantly improved. Recommended implementation direction The core idea is simple: add an optional "knowledge base context" configuration item in the project settings, and when the user enables it, the relevant knowledge is injected into the system prompt when generating questions and answers. There are two recommended ways to access, both of which are not expensive: Method 1: directly read the .md folder Users specify a local directory path (e.g. the wiki/ subdirectory of the Obsidian vault), and Easy Dataset reads the relevant .md file content on demand, parses the [[wikilink]] reference relationships, and inserts the context text into the prompt when generating. This approach is zero-dependency and purely file-based, and is the simplest implementation. Method 2: calling the Obsidian [Local REST API] (https://GitHub.com/coddingtonbear/obsidian-local-rest-api) plugin Obsidian has an official community plugin called the [Local REST API] (https://GitHub.com/coddingtonbear/obsidian-local-rest-api) (installed by more than 300,000 people), which starts an HTTPS service locally, allowing search of note content, reading of files, and querying of link relationships. Users only need to install this plugin in Obsidian, and Easy Dataset can call it via HTTP — no need for file system permissions, and it naturally supports Obsidian's search syntax. This approach provides a better user experience for Obsidian users, especially in scenarios that require fuzzy searches and context retrieval by keyword. Both methods can be supported in parallel, and users can select one in the settings, without interfering with each other. Minimum viable version (avoid overdesign) There is no need to start out with a complex implementation. The minimum version only needs: 1. Add a "knowledge base context" switch and path/API address configuration in the project settings 2. When generating, retrieve the most relevant sections from the knowledge base based on the current slice's keywords 3. Add these sections as additional context to the system prompt The minimum version does not require vector retrieval or embedding, and simple matching based on keywords is already much stronger than the current version. Use case examples I am working on a LLM fine-tuning dataset in the veterinary field, and I maintain a knowledge graph for the diagnosis of avian diseases in Obsidian (the original documents are in the raw/ folder, the organized notes are in the wiki/ folder, and an index.md file serves as a global index). Now, when generating QA, LLM has no idea about the relationship between "chicken infectious fowlpox" and "immune suppression" (chicken infectious fowlpox causes severe immune suppression in chickens), and the generated questions only stay at the level of symptom descriptions, failing to touch on the diagnostic inference chain. If we can inject the relevant entries in the wiki, the quality of the generated dataset will be completely different.

内容来源: ConardLi/easy-dataset