Bug: Incorrect CORRECTNESS_PROMPT — missing format placeholders (user_question, report, answer)

Author: n-tateiwaCreated Nov 13, 2025Updated Feb 12, 2026

Hi team! Thank you for the great project.

I found that CORRECTNESS_PROMPT does not use user_question, report, or answer in tests/prompts.py:

https://github.com/langchain-ai/open_deep_research/blob/b419df8d33b4f39ff5b2a34527bb6b85d0ede5d0/tests/prompts.py#L95-L104

However, the eval_correctness function in tests/evaluators.py formats the prompt with these fields:

https://github.com/langchain-ai/open_deep_research/blob/b419df8d33b4f39ff5b2a34527bb6b85d0ede5d0/tests/evaluators.py#L112

As a result, format() has no effect and the evaluator receives a prompt that lacks the actual question, the generated report, and the authoritative answer.

Could you please confirm whether this is a bug?

Source: langchain-ai/open_deep_research