[Feature Request]: GraphRAG & RAPTOR Scalability Overhaul: Vector ANN Blocking & Global Clustering

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (Language Policy).
• Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
• Please do not modify this template :) and fill in all the required fields.

Is your feature request related to a problem?

Currently, two key knowledge compilation and distillation subsystems in RAGFlow encounter significant scalability and semantic quality bottlenecks when handling medium-to-large corpora:

1. GraphRAG Entity Resolution $O(N^2)$ Memory & CPU Blowup (rag/graphrag/entity_resolution.py):

   • In rag/graphrag/entity_resolution.py, candidate pairing uses itertools.combinations(v, 2) across all nodes of each entity_type.
   • For $N = 10,000$ entities of the same type, this generates $\binom{N}{2} \approx 50,000,000$ comparisons in Python memory.
   • Candidate filtering currently relies on Levenshtein.distance(a, b) <= min(len(a), len(b)) // 2. This fails completely on acronyms and semantic aliases (e.g. "WHO" vs "World Health Organization", where length difference alone causes immediate rejection before any LLM verification can occur).

2. RAPTOR 1D-Watershed Semantic Flaw (rag/advanced_rag/knowlege_compile/raptor.py & internal/ingestion/component/knowledge_compiler/tree/raptor.go):

   • In raptor.py (_get_clusters_ahc) and Go's tree/watershed.go, chunk clustering is performed solely over adjacent chunks along a 1D sequence (adj_sims = np.sum(normalized[:-1] * normalized[1:], axis=1)).
   • This functions as linear text chunk segmentation rather than true semantic clustering. Chunks discussing the same topic in distant sections of a document (e.g., Section 1 and Section 8) can never be clustered or summarized together, defeating the core design of RAPTOR (Sarthi et al., 2024).

Describe the feature you'd like

A scalability and architectural overhaul across the Entity Resolution Engine and RAPTOR Knowledge Compiler:

1. Entity Resolution Engine

• In-Memory Vector ANN Blocking:
  • Replace itertools.combinations with in-memory HNSW / FAISS approximate nearest neighbor index blocking over entity embeddings.
  • Query top-$k$ nearest neighbors for active subgraph entities, reducing candidate generation from $O(N^2)$ to $O(N \log N)$ or $O(N \cdot k)$.
• Acronym Expansion & Heuristics:
  • Add abbreviation extraction (e.g. initialisms "World Health Organization" $\to$ "WHO", parenthetical alias mining) prior to matching.
• Confidence-Tiered Resolution:
  • Fast-path auto-merge for high-confidence vector matches ($\text{cosine similarity} \ge 0.98$ and exact normalized matches) to bypass expensive LLM calls.
  • Send only ambiguous pairs ($0.75 \le \text{sim} < 0.98$) to the LLM verification prompt in batches.
• Index & Storage Optimization (rag/graphrag/general/index.py, rag/graphrag/utils.py):
  • Pre-warm entity embedding Redis cache before candidate resolution to enable fast ANN index construction.
  • Enhance checkpointing and timeout handling to prevent task failures during graph merges.

2. Hierarchical Knowledge Compilation (RAPTOR)

• Global Semantic Clustering (rag/advanced_rag/knowlege_compile/raptor.py):
  • Replace the 1D adjacent-cosine watershed (_get_clusters_ahc) with global clustering.
  • Option A: Gaussian Mixture Models (GMM) with BIC-based optimal $k$ selection over UMAP dimensionality-reduced embeddings (allowing soft/overlapping clusters as in the original RAPTOR paper).
  • Option B: HDBSCAN over UMAP embeddings for density-based variable cluster sizing without fixing $k$.
• Dual-Runtime Harmonization (internal/ingestion/component/knowledge_compiler/tree/raptor.go):
  • Harmonize the Go ingestion tree compiler with the global clustering approach (e.g., global Spherical K-Means / Cosine Agglomerative clustering) to maintain algorithm parity with the Python runtime.

Describe implementation you've considered

• Entity Resolution:
  • Using hnswlib or faiss-cpu for fast in-memory indexing per entity type.
  • Pre-warming embeddings using _batch_embed_cache_misses in utils.py so vectors are ready in Redis prior to resolution.
• RAPTOR Clustering:
  • Re-enabling scikit-learn (and optionally umap-learn) in pyproject.toml for GMM/BIC fitting.
  • For Go, implementing global cosine distance clustering or spherical k-means in internal/ingestion/component/knowledge_compiler/tree/ to replace watershed.go.

Documentation, adoption, use case

• Use Cases:
  • Large-scale knowledge graph construction where entities exceed 5,000–50,000 nodes without worker OOM or timeout.
  • Long document knowledge compilation (research papers, legal briefs, technical specifications) where recurring themes across non-adjacent sections need unified hierarchical summaries.

Additional information

Affected files:

• rag/graphrag/entity_resolution.py
• rag/graphrag/general/index.py
• rag/graphrag/utils.py
• rag/advanced_rag/knowlege_compile/raptor.py
• internal/ingestion/component/knowledge_compiler/tree/raptor.go
• internal/ingestion/component/knowledge_compiler/tree/watershed.go
• pyproject.toml