Proposal: pluggable vector memory backend and optional Milvus support

Author: zc277584121Created Jul 15, 2026Updated Jul 15, 2026

Hi maintainers,

I was looking at the current RAG memory path and wanted to ask whether you would be open to a small vector backend abstraction plus optional Milvus support.

From the current code, autoagent/memory/rag_memory.py appears to build directly on ChromaDB's PersistentClient. Chroma is a good local default, but a narrow backend interface could let users keep the same AutoAgent memory API while choosing different vector stores for different deployment sizes.

A possible scoped shape:

  1. Keep ChromaDB as the default backend.
  2. Introduce a small backend interface around document add/update/delete/search and metadata filtering.
  3. Add an optional Milvus backend behind configuration and an optional dependency.
  4. Support Milvus Lite for local development, and Milvus server / Zilliz Cloud for larger deployments.
  5. Add focused tests that cover backend selection and retrieval behavior without changing the higher-level agent flow.

Would this direction fit AutoAgent? If yes, would you prefer this as an optional backend in core, or as an external adapter/example package?