Knowledge Base: add a standalone recall test for retrieval tuning

Author: Bruce-YiiCreated Sep 10, 2026Updated Sep 10, 2026

Is your feature request related to a problem? Please describe.

When tuning a knowledge base, users currently need to go through an Agent or Workflow debugging flow to inspect what was actually retrieved. That works for debugging a complete downstream request, but it is relatively heavy when the goal is only to answer a narrower question:

For this knowledge base and this query, which chunks are retrieved under a given retrieval configuration, with what scores and sources?

There are several related community signals:

  • In #2218, a user directly suggested adding a recall/retrieval test for evaluating a knowledge base.
  • #2334 and #2459 report difficulty diagnosing poor chunking / retrieval quality.
  • #1857 reports large differences between direct Milvus retrieval and Coze Workflow retrieval under apparently similar inputs.

The problem is not that Coze has no retrieval observability at all. Agent/Workflow debugging can already show recalled chunks. The missing piece is a lightweight knowledge-base-level retrieval tuning loop that does not require first constructing or entering a downstream orchestration flow.

Describe the solution you'd like

Add a standalone Recall Test / Retrieval Playground on the Knowledge Base detail page.

A minimal version could:

  1. Let the user enter a single query.
  2. Let the user provide temporary, test-scoped retrieval settings such as search strategy, TopK, and MinScore.
  3. Reuse the same production knowledge retrieval path rather than introducing a separate retrieval implementation.
  4. Show ranked retrieved chunks with at least final score and source document information.
  5. Make zero-result cases explicit together with the active test settings.
  6. Keep all test settings ephemeral: running a test should not modify the Knowledge Base itself or any Agent/Workflow configuration.

For a first version, I would intentionally keep the scope small: no batch dataset evaluation, no Recall@K / MRR / NDCG metrics, no experiment history, and no retrieval algorithm changes.

Describe alternatives you've considered

  • Use Agent/Workflow debugging only. This is useful for end-to-end debugging, but it mixes knowledge retrieval diagnosis with downstream orchestration and makes quick retrieval tuning slower.
  • Improve retrieval quality directly. That is still important, but repeated reports such as #1857 / #2334 are difficult to diagnose when users cannot isolate query → retrieval result → score → configuration at the Knowledge Base layer.
  • Expose more fields only in the Workflow knowledge-retrieval node. #2253 is a useful historical example: maintainers noted that exposing more retrieval data requires coordinated frontend and backend behavior. A Knowledge-level test surface seems cleaner for this specific tuning workflow than only extending downstream node output.

Additional context

The current codebase already appears to have much of the underlying foundation:

  • the knowledge retrieval domain path returns retrieved slices with scores;
  • the Knowledge Base has its own IDE/detail surface where a test action could naturally live;
  • the public frontend still contains an autogenerated RetrieveTestReq/Resp contract and a generated client path for /api/devops/knowledge_platform/v1/retrieve_test originating from the earlier Bytedance code mirror.

I am not assuming that the old generated RetrieveTest contract is intended to be restored in the open-source product; it may be an internal-only historical artifact. I am mentioning it only because it suggests that this workflow has been modeled before.

Before implementing, I would like to align with the maintainers on three points:

  1. Is a standalone Knowledge-level retrieval test something you want in the open-source product, or should retrieval inspection intentionally remain inside Agent/Workflow debugging?
  2. Is the existing generated RetrieveTest contract still conceptually relevant, or should an OSS-native API be designed around the current knowledge retrieval domain path?
  3. If the direction is welcome, which Knowledge UI surface and API boundary would you prefer for a minimal first version?

If this direction fits the project, I would be happy to take ownership of the scoped implementation after we align on the boundary.