Knowledge Base: add a standalone recall test for retrieval tuning
Is your feature request related to a problem? Please describe.
When tuning a knowledge base, users currently need to go through an Agent or Workflow debugging flow to inspect what was actually retrieved. That works for debugging a complete downstream request, but it is relatively heavy when the goal is only to answer a narrower question:
For this knowledge base and this query, which chunks are retrieved under a given retrieval configuration, with what scores and sources?
There are several related community signals:
- In #2218, a user directly suggested adding a recall/retrieval test for evaluating a knowledge base.
- #2334 and #2459 report difficulty diagnosing poor chunking / retrieval quality.
- #1857 reports large differences between direct Milvus retrieval and Coze Workflow retrieval under apparently similar inputs.
The problem is not that Coze has no retrieval observability at all. Agent/Workflow debugging can already show recalled chunks. The missing piece is a lightweight knowledge-base-level retrieval tuning loop that does not require first constructing or entering a downstream orchestration flow.
Describe the solution you'd like
Add a standalone Recall Test / Retrieval Playground on the Knowledge Base detail page.
A minimal version could:
- Let the user enter a single query.
- Let the user provide temporary, test-scoped retrieval settings such as search strategy, TopK, and MinScore.
- Reuse the same production knowledge retrieval path rather than introducing a separate retrieval implementation.
- Show ranked retrieved chunks with at least final score and source document information.
- Make zero-result cases explicit together with the active test settings.
- Keep all test settings ephemeral: running a test should not modify the Knowledge Base itself or any Agent/Workflow configuration.
For a first version, I would intentionally keep the scope small: no batch dataset evaluation, no Recall@K / MRR / NDCG metrics, no experiment history, and no retrieval algorithm changes.
Describe alternatives you've considered
- Use Agent/Workflow debugging only. This is useful for end-to-end debugging, but it mixes knowledge retrieval diagnosis with downstream orchestration and makes quick retrieval tuning slower.
- Improve retrieval quality directly. That is still important, but repeated reports such as #1857 / #2334 are difficult to diagnose when users cannot isolate query → retrieval result → score → configuration at the Knowledge Base layer.
- Expose more fields only in the Workflow knowledge-retrieval node. #2253 is a useful historical example: maintainers noted that exposing more retrieval data requires coordinated frontend and backend behavior. A Knowledge-level test surface seems cleaner for this specific tuning workflow than only extending downstream node output.
Additional context
The current codebase already appears to have much of the underlying foundation:
- the knowledge retrieval domain path returns retrieved slices with scores;
- the Knowledge Base has its own IDE/detail surface where a test action could naturally live;
- the public frontend still contains an autogenerated
RetrieveTestReq/Respcontract and a generated client path for/api/devops/knowledge_platform/v1/retrieve_testoriginating from the earlier Bytedance code mirror.
I am not assuming that the old generated RetrieveTest contract is intended to be restored in the open-source product; it may be an internal-only historical artifact. I am mentioning it only because it suggests that this workflow has been modeled before.
Before implementing, I would like to align with the maintainers on three points:
- Is a standalone Knowledge-level retrieval test something you want in the open-source product, or should retrieval inspection intentionally remain inside Agent/Workflow debugging?
- Is the existing generated
RetrieveTestcontract still conceptually relevant, or should an OSS-native API be designed around the current knowledge retrieval domain path? - If the direction is welcome, which Knowledge UI surface and API boundary would you prefer for a minimal first version?
If this direction fits the project, I would be happy to take ownership of the scoped implementation after we align on the boundary.
Source: coze-dev/coze-studio