Implement DeepResearchGym-style evaluations

The Problem

In order to improve research quality of the GPT-researcher, there need to be a way to properly measure research quality. Although there are existing evaluation modules in this project, they are misaligned with the goals on research quality (comprehensive, unbiased, and factual), thus cannot provide much guidance for research quality improvements.

The proposal

Replacing existing evaluation implementations with DeepResearchGym evaluation protocol.

And use Researchy Questions, DeepResearch Bench, and ServiceNow DRBench as the baseline to evaluate research quality.

Potential Future Projects

Quality optimization would be enabled once the improved evaluation is setup. Some ideas:

• Use GEPA + FewshotsBootstrap (from DSPy) to optimize prompts
• Support community curated datasets for targeted optimization
• Add feature for user to optimize their GTPR instance with their own data

Appendix: misalignment of existing eval modules

SimpleQAEval is judging whether the research outcome contains the exact answer of a specific, close-ended question. But the point of research is not to find answer to a closed-ended question, but more about factual, unbiased and comprehensive exploration of information.

Existing hallucination eval checks hallucination solely based on the research query and the generated report, because the LLM judge lacks access to the underlying ground truth data, the evaluation result can not really reflect the potential bias/truthfulness of the research outcome.