DPO chat_template Strategy Drops tool_calls and Never Passes tools to the Chat Template

Please check that this issue hasn't been reported before.

• I searched previous Bug Reports didn't find any similar reports.

Expected Behavior

Using rl: dpo with type: chat_template.default on an OpenAI-format tool-calling dataset (tools + messages with tool_calls / tool roles + chosen / rejected) — the common format for multi-step tool-use preference data, where the shared context contains assistant tool_calls turns and tool result turns — should render the same way SFT's chat_template strategy does:

• the tools list is passed to apply_chat_template so tool definitions are rendered into the system prompt
• assistant tool_calls turns are preserved in the rendered prompt
• messages with content: null (standard for OpenAI-format tool call turns) are handled
• a chosen/rejected response that is itself a tool call is extracted correctly
• reasoning_content in chosen/rejected renders into the template's thinking block

Current behaviour

I hit this while training tool calling with DPO on my own OpenAI-format dataset. src/axolotl/prompt_strategies/dpo/chat_template.py copies only role and content from each message, so:

1. assistant tool_calls are silently dropped — the prompt renders an empty assistant turn, and the model trains on corrupted context (no error, easy to miss)
2. tools is never passed to apply_chat_template — tool definitions never reach the system prompt
3. content: null crashes while rendering the chat template — with qwen3_5: TypeError: 'NoneType' object is not iterable (the template iterates non-string content); other templates fail similarly (e.g. can only concatenate str (not "NoneType") to str)
4. if chosen/rejected is itself a tool call (empty content), the response extraction (result["chosen"].find(chosen["content"])) fails and leaks the internal [[dummy_message]] prompt into the training text
5. similarly, reasoning_content in a chosen/rejected response is silently dropped by the content-based extraction — for thinking templates (e.g. qwen3_5) the response should continue the <think> block that the generation prompt opens

Repro rendering (qwen3_5 template, content: "" to avoid the crash):

<|im_start|>system
You are a helpful weather assistant.<|im_end|>      <-- tools never rendered
<|im_start|>user
What's the weather in Paris?<|im_end|>
<|im_start|>assistant
<|im_end|>                                          <-- tool_calls silently dropped
<|im_start|>user
<tool_response>
22C, sunny
</tool_response><|im_end|>
<|im_start|>assistant
<think>

Related: #3217 fixed the KeyError: 'tool' for tool roles, and #3228 adds tool to message_property_mappings — but neither preserves tool_calls nor renders tools. The SFT chat_template strategy already supports all of this (ChatTemplateStrategy._get_tools, transform_message); the DPO strategy is missing feature parity.

Steps to reproduce

1. Create data.jsonl (OpenAI chat format). This is the typical shape when building DPO pairs for multi-step tool use — e.g. from agent trajectories, or by converting SFT tool-calling datasets (ToolACE, hermes-function-calling) into preference pairs; same shape as the dataset in #3217:

{
  "tools": [{"type": "function", "function": {"name": "get_weather", "description": "Get current weather for a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}],
  "messages": [
    {"role": "system", "content": "You are a helpful weather assistant."},
    {"role": "user", "content": "What's the weather in Paris?"},
    {"role": "assistant", "content": null, "tool_calls": [{"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}}]},
    {"role": "tool", "tool_call_id": "call_1", "content": "22C, sunny"}
  ],
  "chosen": {"role": "assistant", "content": "It's 22C and sunny in Paris.", "reasoning_content": "The tool returned 22C and sunny, so I should report that."},
  "rejected": {"role": "assistant", "content": "I cannot check the weather.", "reasoning_content": "I don't have access to weather tools."}
}

1. Run axolotl preprocess config.yaml --debug with the config below
2. Observe the crash during "Mapping RL Dataset":

File "/axolotl/src/axolotl/prompt_strategies/dpo/chat_template.py", line 96, in transform_fn
  result["prompt"] = tokenizer.apply_chat_template(
...
File "<template>", line 85, in top-level template code
TypeError: 'NoneType' object is not iterable

1. With "content": "" instead of null, preprocessing succeeds but silently produces the corrupted prompt shown above (no tools, empty assistant turn, no reasoning in responses)

Config yaml

base_model: Qwen/Qwen3.5-27B
rl: dpo
chat_template: qwen3_5
datasets:
  - path: data.jsonl
    type: chat_template.default
micro_batch_size: 1
gradient_accumulation_steps: 1
learning_rate: 5e-6
output_dir: ./outputs/dpo-out

Possible solution

Mirror the SFT chat_template strategy in dpo/chat_template.py:

• preserve the message properties the chat template actually uses (tool_calls, tool_call_id, name, reasoning_content, …) via JinjaTemplateAnalyzer, skipping None values; union in the standard OpenAI message keys since some templates access properties via message.get(...) which template analysis can't see (e.g. gemma4)
• read field_tools (default "tools") from the sample, decode JSON-encoded strings, and pass tools= to apply_chat_template
• decode JSON-string tool_call.function.arguments into dicts (templates apply | tojson)
• extract chosen/rejected by stripping the rendered dummy-user prompt prefix (longest common prefix) instead of searching for the response content, so contentless tool-call responses and reasoning_content responses extract cleanly
• pass the config's chat_template_kwargs (e.g. enable_thinking) to apply_chat_template, matching SFT

I have a working patch with tests, verified against qwen_25 / qwen3 / qwen3_5 / gemma4 / exaone4 / llama4 / llama3_2_vision / command_a_tool_use / jamba — PR incoming.

Which Operating Systems are you using?

• Linux
• macOS
• Windows

Python Version

3.12

axolotl branch-commit

main/6401a353

Acknowledgements

• My issue title is concise, descriptive, and in title casing.
• I have searched the existing issues to make sure this bug has not been reported yet.
• I am using the latest version of axolotl.
• I have provided enough information for the maintainers to reproduce and diagnose the issue.