[Bug] Blinko 1.8.8 PDF attachment is indexed but attachment text is not passed into RAG context

# [Bug] Blinko 1.8.8 PDF attachment is indexed but attachment text is not passed into RAG context

## 中文说明

### 环境

- Blinko 版本：1.8.8
- 部署方式：Docker
- 测试类型：纯文本 PDF 附件
- 向量数据库：`/app/.blinko/vector/embeddings.db`

---

## 问题描述

在 Blinko 1.8.8 中，我测试了 PDF 附件的 AI 检索能力。

现象是：

```text
PDF 上传成功
→ PDF 可以被解析
→ Embedding 成功
→ PDF 正文已经进入向量数据库
→ 但 AI 无法根据 PDF 独有内容回答问题

为了避免笔记正文干扰测试，我只在 PDF 中放置测试信息，笔记正文不包含这些内容。

预期结果：

AI 应该能够根据 PDF 附件正文回答问题。

实际结果：

AI 返回 “未找到”。

第一个发现的问题：缺少 pdf-parse

最初在重建 embedding 时出现：

ERR_MODULE_NOT_FOUND
Cannot find package 'pdf-parse' imported from /app/node_modules/@langchain/community/dist/document_loaders/fs/pdf.cjs

Failed to load pdf-parse.
Please install it with eg. npm install pdf-parse.

在容器内执行：

cd /app
npm install pdf-parse

之后，PDFLoader 可以正常读取 PDF 正文。

验证 PDFLoader

测试 PDF 内容：

Project Code: MORNINGSTAR-9417
Device: Pine-Terminal
Unique ID: MX-274681
Owner Code: BIRCH-63
Target Date: 2032-08-26

直接调用 PDFLoader 后可以成功输出完整正文。

因此可以确认：

PDF 文件正常
PDFLoader 正常
pdf-parse 正常

验证 PDF 已成功进入向量数据库

向量数据库：

/app/.blinko/vector/embeddings.db

查询测试内容：

sqlite3 /app/.blinko/vector/embeddings.db \
"SELECT id, vector_id, substr(metadata,1,1000)
 FROM blinko
 WHERE metadata LIKE '%MX-274681%';"

可以找到对应记录。

metadata 中包含类似：

{
  "text": "Project Code: MORNINGSTAR-9417 ... Unique ID: MX-274681 ...",
  "isAttachment": true,
  "noteId": 12
}

因此可以确认：

PDF 正文已经成功解析、Embedding，并写入向量数据库。

第二个发现的问题：附件正文没有进入最终 RAG 上下文

继续检查 Blinko 1.8.8 的 queryVector() 逻辑后发现：

向量查询结果中可以包含附件向量，但是最终传给 LLM 的 aiContext 只由笔记正文构成。

原逻辑类似：

h=p.map(f=>f.content+`\n`)||""

也就是说：

PDF attachment
→ parsed successfully
→ embedded successfully
→ vector search can retrieve it
→ attachment metadata.text is available
→ but metadata.text is not added to aiContext
→ LLM cannot see PDF content
→ AI returns "not found"

临时修复

我临时修改了 RAG context 构造逻辑，将命中的附件 metadata.text 也加入 aiContext。

核心思路：

note.content
+
matched attachment metadata.text

修改后，重新进行相同测试。

问题：

What is the Unique ID of MORNINGSTAR-9417?

修复前：

Not found

修复后：

MX-274681

进一步验证

我还使用了一份更复杂的 PDF，包含：

• 多个 section
• 设备信息
• 风险等级
• 项目阶段
• 金额
• 条件依赖
• 跨段信息

然后提出需要跨多个段落综合判断的问题。

修复后 Blinko 可以正确完成复杂 PDF 问答，说明问题不只是单个关键词检索，而是附件正文是否真正进入最终 LLM context。

当前确认的结论

目前在我的 Blinko 1.8.8 Docker 环境中确认：

1. PDF 解析最初会因为缺少 pdf-parse 失败；
2. 安装 pdf-parse 后，PDF 正文可以成功读取；
3. PDF 正文可以成功写入 /app/.blinko/vector/embeddings.db；
4. 原始 RAG 逻辑可以检索到附件向量；
5. 但附件 metadata.text 没有被加入最终 aiContext；
6. 将附件正文加入 aiContext 后，AI 可以正确回答 PDF 内容。

建议

建议考虑：

1. 确认 Docker / production runtime 是否应该默认包含 pdf-parse；
2. 检查 queryVector() 对 attachment vector 的处理；
3. 将命中的 attachment metadata.text 纳入最终 RAG context；
4. 增加一个 PDF-only attachment retrieval 的 regression test；
5. 测试场景应确保：
   • 测试答案只存在于 PDF；
   • 笔记正文不包含答案；
   • 验证 embedding；
   • 验证 vector hit；
   • 验证最终 LLM context。

如果需要，我可以继续提供：

• 最小测试 PDF；
• 查询数据库的命令；
• 临时 patch；
• 修复前后的截图；
• 自定义 Docker 镜像构建方式。

English

Environment

• Blinko version: 1.8.8
• Deployment: Docker
• Test type: text-based PDF attachment
• Vector database: /app/.blinko/vector/embeddings.db

Problem

I tested PDF attachment retrieval in Blinko 1.8.8.

The observed behavior is:

PDF uploads successfully
→ PDF can be parsed
→ embedding succeeds
→ PDF text exists in the vector database
→ but AI still cannot answer questions based on PDF-only content

To avoid contamination from note content, the test data exists only inside the PDF attachment. The note body does not contain the answer.

Expected:

AI should answer based on the PDF attachment content.

Actual:

AI returns "not found".

First issue found: missing pdf-parse

Initially, rebuilding embeddings produced:

ERR_MODULE_NOT_FOUND
Cannot find package 'pdf-parse' imported from /app/node_modules/@langchain/community/dist/document_loaders/fs/pdf.cjs

Failed to load pdf-parse.
Please install it with eg. npm install pdf-parse.

Inside the container, I ran:

cd /app
npm install pdf-parse

After that, PDFLoader was able to extract PDF text successfully.

PDFLoader verification

Test PDF content:

Project Code: MORNINGSTAR-9417
Device: Pine-Terminal
Unique ID: MX-274681
Owner Code: BIRCH-63
Target Date: 2032-08-26

Directly calling PDFLoader returned the complete text.

So the following were confirmed:

PDF file is valid
PDFLoader works
pdf-parse works

Verified that PDF content is stored in the vector database

Vector database:

/app/.blinko/vector/embeddings.db

Query:

sqlite3 /app/.blinko/vector/embeddings.db \
"SELECT id, vector_id, substr(metadata,1,1000)
 FROM blinko
 WHERE metadata LIKE '%MX-274681%';"

The record can be found.

The metadata contains data similar to:

{
  "text": "Project Code: MORNINGSTAR-9417 ... Unique ID: MX-274681 ...",
  "isAttachment": true,
  "noteId": 12
}

This confirms that the PDF content was successfully:

parsed
→ embedded
→ stored in the vector database

Second issue found: attachment text is not included in final RAG context

After inspecting Blinko 1.8.8 queryVector() logic, I found that attachment vectors can be returned by vector search, but the final aiContext is built only from note content.

The original logic is similar to:

h=p.map(f=>f.content+`\n`)||""

This means:

PDF attachment
→ parsed successfully
→ embedded successfully
→ vector search can retrieve it
→ attachment metadata.text exists
→ metadata.text is not added to aiContext
→ LLM cannot see the PDF content
→ AI returns "not found"

Temporary fix

I modified the RAG context construction so that matched attachment metadata.text is also included in aiContext.

Conceptually:

note.content
+
matched attachment metadata.text

Then I repeated the same test.

Question:

What is the Unique ID of MORNINGSTAR-9417?

Before the fix:

Not found

After the fix:

MX-274681

Additional verification

I also tested a more complex PDF containing:

• multiple sections
• device information
• risk levels
• project phases
• budget values
• dependency conditions
• cross-section information

Then I asked a question requiring reasoning across multiple sections.

After the fix, Blinko was able to answer correctly.

This suggests the issue is not only keyword retrieval, but whether attachment text is actually included in the final LLM context.

Confirmed findings

In my Blinko 1.8.8 Docker environment, I confirmed:

1. PDF parsing initially fails because pdf-parse is missing;
2. after installing pdf-parse, PDF text extraction works;
3. PDF content is successfully stored in /app/.blinko/vector/embeddings.db;
4. attachment vectors can be retrieved;
5. attachment metadata.text is not included in the final aiContext;
6. after including attachment text in aiContext, PDF QA works correctly.

Suggested fix

Please consider:

1. verifying whether pdf-parse should be included in the Docker / production runtime dependencies;
2. reviewing attachment handling inside queryVector();
3. including matched attachment metadata.text in the final RAG context;
4. adding a regression test for PDF-only attachment retrieval;
5. ensuring the regression test verifies:
   • answer exists only in the PDF;
   • note body does not contain the answer;
   • embedding succeeds;
   • vector retrieval succeeds;
   • attachment text reaches the final LLM context.

If useful, I can provide:

• a minimal reproducible PDF;
• database inspection commands;
• the temporary patch;
• before/after screenshots;
• the custom Docker image build method.

<img width="1175" height="800" alt="Image" src="https://github.com/user-attachments/assets/4efe99d4-a541-4ff2-879e-eb9f51783463" />

<img width="535" height="733" alt="Image" src="https://github.com/user-attachments/assets/ff8bdd1e-38bb-496c-9c46-72ce59b21558" />
、

<img width="775" height="761" alt="Image" src="https://github.com/user-attachments/assets/c6a7cb3a-7211-4645-9067-409b1cb03b9c" />

<img width="1284" height="2778" alt="Image" src="https://github.com/user-attachments/assets/267bdf46-759b-4470-bbb3-ea4fb5348d0e" />

<img width="2005" height="657" alt="Image" src="https://github.com/user-attachments/assets/458b40bd-4fc9-4793-be46-210b2f9b2146" />