[DeepPlanning] LLM report→JSON conversion silently drops drafts and copies the few-shot budget (2400/…/6700), inflating travelplanning scores

中文摘要：DeepPlanning travelplanning 的「方案原文 → JSON」转换由 qwen-plus 完成。转换提示没有规定「原文有多份草稿」「原文没写预算」时怎么办，示例本身还前后矛盾，于是转换模型会静默挑一份草稿、把 few-shot 示例里的预算数字（2400/2000/1500/500/300/6700）填进去。评分器只看 JSON，所以这些被转换器「修好」的缺陷不会被扣分，分数偏高。下面附最小复现（3/3 触发）和一段补充提示（3/3 修好）。

Summary

In benchmark/deepplanning/travelplanning, the agent's final plan text is converted to JSON by an LLM (qwen-plus) before scoring. The conversion prompt says the conversion must be lossless with no additions or omissions (prompts.py#L408). In practice the converter silently rewrites the plan in two ways:

1. It keeps only one draft. When the report contains several drafts (the agent rewrote the itinerary and submitted all versions), the converter keeps only the last one.
2. It fills in the budget from the few-shot example. When the report has no Budget Summary section, the converter copies the example's numbers 2400 / 2000 / 1500 / 500 / 300 / 6700 into budget_summary.

The evaluator only sees the JSON, so real agent failures (never converging on one plan, never computing a budget) are hidden and can even earn points. Note that 6700 often lands inside the cost-accuracy tolerance.

Minimal reproduction (3/3)

Input: a 2-day plan written twice. Draft 1 visits 灵隐寺 on day 2, draft 2 visits 中国茶叶博物馆. There is no budget section.

Day 1:
Current City: from 上海 to 杭州
Accommodation: 杭州西湖国宾馆，￥880/间/晚
08:00-09:10 | travel_intercity_public | train G7311，上海虹桥站 - 杭州东站，￥73
09:10-09:40 | buffer | 下火车、取行李
09:40-10:20 | travel_city | 杭州东站 - 杭州西湖国宾馆，12km，40min，￥45
10:20-11:00 | hotel | 办理入住，杭州西湖国宾馆
11:00-12:00 | meal | 午餐，楼外楼（孤山路店），￥150/人
12:30-16:30 | attraction | 西湖风景名胜区，￥0
18:00-19:00 | meal | 晚餐，知味观（湖滨店），￥90/人
19:30-24:00 | hotel | 休息，杭州西湖国宾馆

Day 2:
Current City: from 杭州 to 上海
Accommodation: -
08:30-11:30 | attraction | 灵隐寺，￥75
12:00-13:00 | meal | 午餐，外婆家（湖滨银泰店），￥70/人
14:00-14:40 | travel_city | 外婆家（湖滨银泰店） - 杭州东站，10km，40min，￥40
15:05-16:10 | travel_intercity_public | train G7358，杭州东站 - 上海虹桥站，￥73

等等，第二天灵隐寺上午人太多，我重新安排一下。

Day 1:
Current City: from 上海 to 杭州
Accommodation: 杭州西湖国宾馆，￥880/间/晚
08:00-09:10 | travel_intercity_public | train G7311，上海虹桥站 - 杭州东站，￥73
09:10-09:40 | buffer | 下火车、取行李
09:40-10:20 | travel_city | 杭州东站 - 杭州西湖国宾馆，12km，40min，￥45
10:20-11:00 | hotel | 办理入住，杭州西湖国宾馆
11:00-12:00 | meal | 午餐，楼外楼（孤山路店），￥150/人
12:30-16:30 | attraction | 西湖风景名胜区，￥0
18:00-19:00 | meal | 晚餐，知味观（湖滨店），￥90/人
19:30-24:00 | hotel | 休息，杭州西湖国宾馆

Day 2:
Current City: from 杭州 to 上海
Accommodation: -
08:30-11:00 | attraction | 中国茶叶博物馆，￥0
12:00-13:00 | meal | 午餐，外婆家（湖滨银泰店），￥70/人
14:00-14:40 | travel_city | 外婆家（湖滨银泰店） - 杭州东站，10km，40min，￥40
15:05-16:10 | travel_intercity_public | train G7358，杭州东站 - 上海虹桥站，￥73

Script (uses the official functions unchanged):

from evaluation.convert_report import process_single_report
from prompts import get_format_convert_prompt
fmt = get_format_convert_prompt('zh')
process_single_report(Path('reports/id_1.txt'), out_dir, 'qwen-plus', client, fmt, threading.Lock(), 1)

Result with qwen-plus (snapshot qwen-plus-2025-07-28, no temperature set, same as the official call), 3 runs:

How common this is on real agent outputs

We looked at 95 reports produced by a Qwen3.5-9B agent (48 queries × 2 rollouts, zh):

• Multiple drafts: 20 of the 95 reports contain two or more drafts. qwen-plus kept only one draft in 20 of 20. We also tried GPT-5.6 Luna, DeepSeek V4.1 Flash and Kimi K3 as converters; each kept only one draft in 20 of 20. So this is a prompt problem, not a problem of one model.
• Example budget copied: in 38 of 95 conversions (40%), budget_summary exactly equals the example's 2400/2000/1500/500/300/6700. Other converters fail differently on the same reports: some fill 0, some sum the itinerary themselves.
• Effect on scores: on 5 hard reports, conversions that keep every draft and leave the missing budget empty average 0.300 composite. The usual draft-picking conversions average 0.388. The difference is score the converter earns for the agent.

Root causes in the conversion prompt / call

1. The prompt gives no rules for multiple drafts, a missing budget section, or placeholder text. "Lossless" alone is not enough; models read the task as "clean this up".
2. The end-to-end example is internally inconsistent. Its output does not match its own input, which teaches the model that rewriting is acceptable:
   • zh: input from 杭州 to 北京, 北京金霖酒店…，￥694/间/晚 (L498-L499); output from 上海 to 北京, 北京王府井文华东方酒店, 1000 (L530-L533). The output values come from the agent-prompt example at L148-L149, so this looks like a copy-paste mix-up.
   • en: same mismatch (from Hangzhou to Beijing / Jinlin Hotel ¥694 in the input vs from Shanghai to Beijing / Mandarin Oriental 1000 in the output, L756-L757 vs L788-L791).
   • The schema says travel_city.details has only from/to/distance/duration/cost (L475-L476). The example output nevertheless adds "mode": "taxi" / "walking" (L558), although the input never mentions a taxi.
3. No temperature is passed (convert_report.py#L141-L145), so the provider default (≈1.0) applies to an extraction task. Re-converting the same 95 reports with qwen-plus gave a different composite score on 6.5% of them (3.2% even at temperature=0).
4. conversion_model = 'qwen-plus' is a rolling alias (convert_report.py#L237). The weights behind it can change, so scores from different dates are not comparable.

Suggested fixes

• Add explicit rules to the conversion prompt. Appending the rules below to the system prompt fixed the reproduction above in 3 of 3 runs: both drafts kept (4 days), budget_summary all null, no invented fields.
  • Keep every draft in order. Do not deduplicate or pick a "final" version. If a literal <plan> tag exists, convert only what follows it.
  • If there is no Budget Summary section, set the numeric budget fields to null. Do not copy the example numbers, do not use 0, do not compute totals.
  • Copy labelled values and names verbatim, and keep placeholder text as is.
  • The example only illustrates the format; its concrete values are not defaults.
• Fix the example so that its output matches its input, and remove mode from travel_city (or add it to the schema).
• Pass temperature=0 and pin a dated model snapshot.
• Note on max_tokens=10240: once drafts are kept, some long multi-draft reports exceed this limit and the conversion is truncated. Either raise the limit, or have the evaluator flag "multiple drafts" / "no budget section" as failures directly instead of relying on the converter.
• Longer term: the plan format is fixed (Day N: / HH:MM-HH:MM | type | ...), so a deterministic parser would remove this LLM-judgement step from the scoring path entirely.

补充规则（优先级高于上文所有示例；与示例冲突时以本节为准）

你是抄写员，不是编辑。输入里有什么就转什么，输入有缺陷就原样保留缺陷。下面几种情况尤其要注意：

1. 不要挑稿，全部保留
   - 输入可能包含多份行程草稿：例如 Day 1–Day 3 写完后，又从 Day 1 重新写了一遍；或者前后两份内容不同的行程。
   - 不要判断哪一份是「最终版」，不要去重，不要合并，不要只保留其中一份。
   - 按原文出现顺序，把所有 Day 段落依次转进 daily_plans。day_number 照抄原文编号，重复也照抄。
   - 唯一例外：如果原文里出现字面的 <plan> 标签，只转 <plan> 之后的内容。标签之后的 Day 段落同样全部保留，即使彼此逐字相同。

2. 不要编数据，原文没写的就不填
   - 原文没有 Budget Summary（预算汇总）段落时，budget_summary 里的各个数值字段一律填 null，currency 可以填 "CNY"。不要照抄上文示例里的 2400 / 2000 / 1500 / 500 / 300 / 6700，不要填 0，也不要自己把行程里的费用加起来。
   - 原文有预算段时，照抄段落里标注的数值。例如原文写「**Meals: 900 元**」，就填 900；即使后面的算式算出别的数，也不要用算出来的数替换。原文有多个预算段时，取最后出现的那一段。
   - 原文没写价格的地方，cost 填 null，不要估一个数。

3. 名称逐字照抄，不要替换或补全
   - 航班号、车次、酒店名、餐厅名、景点名逐字照抄，不删前缀后缀（例如「东航MU5101」不能写成「MU5101」），原文有错别字也照抄。
   - 原文是占位文字时（例如「周边小吃」「待定」「某某酒店」），名称字段照抄这段占位文字，不要换成一个具体名字。
   - 原文写「休息（某城市名）」这类只有城市名、没有酒店名的住宿，name 照抄原文文字，不要把城市名当成酒店名。

4. 上文的完整示例只用来示范格式
   - 示例里的具体城市、酒店、价格、交通方式都不是默认值，不要带进输出。
   - travel_city 的 details 只包含 Schema 规定的 from / to / distance / duration / cost 五个字段，不要添加 mode。

Caveat: with this exact wording, qwen-plus sometimes over-applies rule 3 and copies the price into the hotel name (e.g. …酒店，￥278/间/晚). Rule 3 should explicitly say that prices still go into cost.