AutoRAG

AutoRAG

A self-evolving librarian agent for document collections.

[!IMPORTANT] Looking for the original AutoRAG (RAG AutoML / pipeline optimization tool)? This repository now hosts AutoRAG 2.0, a complete reimagining of AutoRAG as a self-evolving librarian agent. The original Python-based AutoRAG — the RAG AutoML tool for automatically finding an optimal RAG pipeline for your data — now lives in the legacy/ directory of this repository.

The legacy AutoRAG is NOT abandoned. It continues to be maintained (bug fixes, dependency updates, and PyPI releases via pip install AutoRAG) in maintenance mode. Existing users can keep using it exactly as before — see the legacy README for its documentation, and file issues in this repository as usual. New feature development is focused on AutoRAG 2.0.

AutoRAG searches your PDFs, wikis, notes, research papers, and knowledge bases — then curates the results into clean, numbered knowledge units. No raw grep dumps. Just answers.

AutoRAG is a customized Pi agent — the Pi agent loop configured into a librarian. The AutoRAG librarian retrieves candidates, reads source files directly, judges the evidence, and curates the structured answer. Its model and provider come from the user's authenticated runtime; AutoRAG does not ship a private provider default.

AutoRAG itself is the specialized search agent, not a coordinator for other model roles. You configure one model, and that model owns the complete retrieval, reading, judgment, and curation loop.

Core values

Three principles drive every design decision:

1. Never migrate your data to search it. AutoRAG federates CLI-owned stores (katok, discrawl, qmd, msgvault, rclone, …) in place. No forced ingestion into a central index, no third-party server holding a copy of your corpus. Results carry source-native identities (/kakao/<instance>/chunks/<chunk>) with scope-checked access — see the competitive landscape study for why this is the durable differentiator.
2. Just works — no RAG degree required. Install it and it works: minimal configuration, no pipeline tuning, no vector-DB operations, no API keys. The local embedder (EmbeddingGemma via Ollama) and MinSync auto-install handle the "RAG plumbing" invisibly.
3. Fast by design. One configured model owns the whole loop; retrieval runs locally over MinSync CDC chunks (BM25 / vector / hybrid); interactive search is optimized for low latency.

Why AutoRAG

The problem with search tools

Every search tool gives you the same thing: a list of file paths and matching lines. Then you have to:

• Open each file
• Read the surrounding context
• Decide what's relevant
• Synthesize an answer
• Remember what worked for next time

That's the human doing all the hard work. The tool just points.

AutoRAG does the hard work

AutoRAG is not a search tool. It's a librarian — it searches, reads, thinks, and reports back:

You ask:  "What were the key findings in the Q3 report?"

AutoRAG:
[1] Revenue grew 23% YoY to $4.2M, driven by enterprise contracts. (pages 3-5)
[2] Three new risk factors: supply chain, regulatory, talent retention. (pages 12-14)
[3] Headcount target missed by 12 — engineering hiring bottleneck. (page 8)

No file paths. No line numbers. Just curated knowledge you can act on.

It gets smarter over time

AutoRAG has a self-evolving memory system. Every search teaches it something:

• Which retrieval methods work for which types of queries
• Which document areas are most productive
• What the caller found useful (via explicit feedback)

A fresh AutoRAG tries everything. A seasoned one knows exactly where to look. This is not a static configuration — it's learned behavior from real usage.

Multiple retrieval methods, one interface

Different documents need different search strategies:

AutoRAG supports pluggable retrieval methods. Local lexical BM25, semantic vector, and hybrid retrieval all go through MinSync over one shared CDC chunk lifecycle, wired through the RetrievalMethodRegistry. The librarian invokes retrieval tools, reads the underlying documents directly through bash, and curates one unified result set after ResultMerger score normalization and deduplication. External datasources keep their own archive/index lifecycle.

BM25, vector, and hybrid are enabled by default whenever MinSync is enabled. Disable local indexing with "minSync": false, or disable only lexical search with "bm25": false. MinSync auto-installs a verified release into <workspace>/.autorag/bin on first use (autoInstall: true); set "autoInstall": false only when managing the binary yourself. Configure minSync.embedder via autorag init --embedder-* flags for remote embedding endpoints, and set minSync.maxChunkSize (or --minsync-max-chunk-size) when a local embedder has a smaller context window. AutoRAG never forces TEI or any external embedding service.

See docs/minsync-setup.md for automatic installation, managed binary paths, and the local EmbeddingGemma QA flow.

Real directory access

AutoRAG reads configured source directories directly through its built-in bash tool. Retrieval tools can supply candidate paths, but the same agent opens the source material before curating. Answers are returned as a structured SearchDocumentsResponse; results carry their real source (file path or datasource id) in the internal mapping for feedback and curation. MinSync indexes parsed markdown mirrors under .autorag for BM25, vector, and hybrid retrieval.

Thin PDF extraction retry

The default PDF parser performs a cheap quality check for multi-page PDFs. When local markdown is unusually sparse (fewer than 800 characters or fewer than 40 characters per detected page, for at least three pages), it retries through OpenDataLoader's docling-fast hybrid backend with hybridMode: "auto" and a 30-second timeout. Dense PDFs are not retried, and hybrid is never used for single-page PDFs or as the first path for images.

The gate is parser-owned and can be tuned through trusted programmatic parserOptions:

new AutoRAGAgent({
  searchPaths: ["/path/to/documents"],
  parserOptions: {
    thinExtract: {
      minPages: 3,
      minChars: 800,
      minCharsPerPage: 40,
      timeoutMs: 30_000,
      hybrid: "docling-fast",
      hybridMode: "auto",
    },
  },
});

If the hybrid sidecar is missing, times out, or fails, AutoRAG keeps the local markdown and emits pdf-extract-thin plus pdf-hybrid-unavailable diagnostics; refresh remains successful.

Default Jikji discovery and indexing

AutoRAG uses Jikji by default as a local CLI-backed find-first discovery and indexing layer. Jikji is not registered as a retrieval backend: it supplies bounded discovery answer packs while the librarian still reads original files directly.

The default agent calls jikji find ROOT "query" --json via the jikji_find tool. The tool parses and validates the upstream answer pack and exposes its handoff_action, tool_call_policy, and agent_should_not_rerank to the librarian. Direct file reading remains available for source verification. prepare/refresh remain for indexing only and do not answer queries directly. If the binary or Rust toolchain is unavailable, Jikji reports a diagnostic and the normal filesystem/retrieval paths continue.

New autorag init configs include "jikji": {}. To opt out, set "jikji": false in config.json; programmatic callers can pass jikji: false.

Programmatic use:

const agent = new AutoRAGAgent({
  searchPaths: ["/path/to/documents"],
});
await agent.prepareJikji();

Duplicate document management with dupey

AutoRAG can use the external dupey CLI to detect exact, near, and containment document families.

autorag duplicates /path/to/documents
autorag duplicates --json

The command is read-only: it reports exact duplicate groups and review guidance, but never moves or deletes source files. The scan_duplicate_documents Agent tool exposes the same read-only scan to the orchestrator.

Exact duplicate exclusion is enabled by default during parsed-mirror refresh. For each exact canonical-text hash, the newest filesystem copy is indexed and older copies are omitted. Disable it in config.json when both copies must be searchable:

{
  "dupey": { "enabled": true },
  "excludeExactDuplicates": false
}

If dupey is not installed or fails, refresh continues without exclusion and reports no destructive action; install it with cargo install dupey.

The same configuration shape customizes Jikji when needed:

{
  "enabled": true,
  "binaryPath": "jikji",
  "timeoutMs": 10000,
  "maxBufferBytes": 1048576,
  "includeHidden": false,
  "includeSensitive": false,
  "maxFiles": 0,
  "writeAgentRules": false,
  "enableMediaIndex": false,
  "exclude": []
}

Call agent.prepareJikji() (or agent.refresh()) to prepare configured source roots. Hidden files, sensitive files, and media indexing are disabled by default; AutoRAG does not pass --include-hidden, --include-sensitive, or --enable-media-index unless the corresponding option is true. AutoRAG-managed prepare runs with --no-agent-rules by default, so it never rewrites the consumer repo's AGENTS.md/CLAUDE.md/.cursorrules; an explicit writeAgentRules: true opt-in re-enables upstream routing-block injection. AutoRAG passes --enable-media-index only when enableMediaIndex: true.

The upstream Rust PrepareArgs defines reference defaults that AutoRAG does not override unless explicitly configured: parse timeout 5.0, max hash bytes 512 MiB, doc text max chars 2,000,000, doc text chunk chars 1,000,000, and media index max MB 25.0. AutoRAG emits --parse-timeout, --max-hash-bytes, --doc-text-max-chars, --doc-text-chunk-chars, and --media-index-max-mb only when the matching option is set, so the upstream defaults apply otherwise. AutoRAG answers queries through jikji find (find-first) plus the Pi agent loop and its registered retrieval methods; prepare/refresh are indexing-only.

Datasource skills

Datasource skills let AutoRAG search external, server-configured data sources through the same retrieval pipeline as local documents. A skill describes what it indexes, how it should be refreshed, what source instances exist, and which permission tags/scopes bound access. Retrieval still flows through RetrievalMethodRegistry → ParallelRetriever → datasource result filtering → ResultMerger; datasource skills do not create a parallel search path.

Every datasource can be registered multiple times through a connection alias: use the config key as the unique name and set type to the reusable backend (mailcrawl, github, slack, discord, kakao, cloud-drive, and so on). Each alias becomes an independently loadable agent skill with its own source scope and workspace namespace. Chat aliases search all channels by default; trusted channels.ids / channels.names allowlists can expose a particular channel or group chat as its own datasource.

Security defaults are intentionally strict:

• datasource access is default-deny unless trusted server/API configuration supplies datasourceAccess.allowedTags; allowedScopes a