terminal-bench-rl

Terminal-Bench-RL: Training Long-Horizon Terminal Agents with Reinforcement Learning

TL;DR:

• I successfully built stable RL training infrastructure that scales to 32x H100 GPUs across 4 bare metal nodes for training long-horizon terminal-based coding agents.
• In doing so, I developed Terminal-Agent-Qwen3-32b to become the highest scoring Qwen3 agent on terminal-bench. WITHOUT training!:
  • Unfortunately I am too GPU poor to train a SOTA coding agent (estimated £30k-£50k in compute required), but if anyone has the GPUs, this project should get you there!

This project builds upon the rLLM framework developed by UC Berkeley Sky Lab, extending it with custom environments and infrastructure specifically designed for terminal-based agent training.

Table of Contents

• Training on $1M worth of compute
  • Other training runs
• Placing a spot on the Terminal Bench Leaderboard
  • ️ Action-Based Architecture
• Training details
  • ⚖️ Reward Design
  • ✅ Answer Verification (65% weight)
  • LLM-as-a-Judge (35% weight)
    • Judge Evaluation System
    • Dynamic LLM Judge Switching
• ️ rLLM Integration Architecture
  • Terminal Agent (TerminalBenchAgent)
  • Docker Environment (DockerIsolatedEnv)
• Training & Rollout Details
  • Rollout Strategy
  • ⚙️ Training Configuration Presets
  • Key Hyperparameters (Production Config)
• ️ Dataset Details
  • Dataset Structure
  • Training Environment Creation
  • Docker Resource Management
  • Dataset Preparation Pipeline
• Getting Started
  • Development Setup
  • Terminal Bench Evaluation Reproduction
  • Training Deployment
    • Single Node Training
    • Multi-Node Training
• Future Improvements
  • Full Training Run
  • Curriculum learning
  • Dataset Expansion
  • Smart Data Filtering
• Acknowledgements

---

Training on $1M worth of compute

This image shows my training code running at full throttle on 32x H100's, distributed across a 4x bare metal node cluster, training Qwen3-32B. Thank you Hyperbolic for such a streamlined experience! This was fun!

Due to the extreme cost of this level of compute, I was not able to run it forever! So I made sure it worked and also ran the code on less extravagent hardware setups too.

Other training runs

I also ran Qwen3-32B training for longer on a 2x bare metal node cluster with 16x H100s:

Also 1 VM instance with 8x H100s:

My longest training run was using 2xA100s on a single VM instance, where I trained Qwen3-8B for over 60 steps:

Note: I did not expect the 8B to begin learning the complex behaviours required to solve the tasks in the dataset. However it was great to run the training through the dataset and ensure the code is stable.

Placing a spot on the Terminal Bench Leaderboard

Terminal bench is a brilliant benchmark created by Stanford and Laude Institute to quantify agents' ability to complete complex tasks in the terminal.

Through prompt engineering & custom tool design, my Qwen3-32B agent outperformed Stanford's Terminus-Qwen3-235B-30A MoE agent, as well as Deepseek R1 & OpenAI's GPT-4.1 with Codex agent, to become the highest scoring Qwen3 agent on the leaderboard.

The results.json for the eval run can be found here.

I am sure that with the compute budget for training, my agent would climb the leaderboard significantly.

Agent details

My motivation behind this entire project was to place on the leaderboard of terminal bench by using RL to train a sophisticated LLM agent. In order to do so, I developed the tools (inspired by Claude Code) which a capable AI agent would use to help complete complex terminal/coding tasks, as well as a system message which encouraged the agent to use those tools and approach the task in a specific way.

These tools can be found here and include:

• ** Todo Management**: Planning and tracking task progress
• ** File Operations**: Read, write, and edit files
• ** Search Tools**: Grep, glob, and ls for file exploration
• ⚡ Bash Execution: Run terminal commands with output capture
• ️ Scratchpad: Space for note-taking
• ** Task Completion**: Signal when the agent believes the task is complete

Note: Technically the agent could have access to only the bash tool and would still have the same capabilites as all these tools above. Saving the development time and maintenance. However by providing clear APIs to specific tools, it enables the agent to understand and leverage tools much more effectively.

️ Action-Based Architecture

The agent communicates through a structured XML/YAML format that ensures reliable parsing and execution:

operations:
  - action: add
    content: "Find and analyze all Python test files"
  - action: add
    content: "Run pytest and fix any failing tests"
view_all: true

cmd: 'find . -name "*.py" -path "*/test*" | head -10'
timeout_secs: 30

This architecture provides:

• Type Safety: Each action (bash, file, search, todo) has a dedicated handler with validation
• Error Recovery: Malformed YAML triggers helpful error messages guiding the agent to correct syntax
• Sequential Execution: Actions are processed one at a time with mandatory stop-and-wait behavior
• Consistent Feedback: Every action returns structured results the agent can learn from and adjust its plan

As well as developing these tools, I also wrote out a system prompt which encourages best practices such as:

• Structured Task Execution: Clear problem approach phases (Planning → Exploration → Execution → Verification)
• Multi-Turn Interaction: Action-environment cycle with proper stop-and-wait behavior
• Mandatory Todo Management: Required initial planning and continuous task tracking
• Read-Only Exploration: Gather information before making any changes

With this system message & tool combination + a capable LLM (I chose Qwen3-32B), I was able to place 19th on the terminal bench leaderboard (currently under submission) with a score of 13.75%. This outperformed:

• Terminus agent with Qwen3-235B by Stanford
• Terminus agent with Deepseek-R1 by Stanford
• Codex agent with GPT-4.1 by OpenAI
• Codex agent with codex-mini by OpenAI

The agent can be seen here.

I would be extremely excited to see where Qwen3-32B would be on the leaderboard if I could afford to pay for the compute cost of a proper RL run!

---

Training details

As mentioned above, the compute costs of a full training run on a 32B LLM for long horizon terminal/coding tasks are not accessible for me, however the training code and dataset is ready to go and has been tested to train stably on hardware setups from 2x A100s all the way to 32x H100s.

⚖️ Reward Design

To provide meaningful supervision during RL, rewards were computed using two complementary methods:

✅ Answer Verification (65% weight)

• Each training datapoint included Python unit tests to verify task completion
• Tests were assigned individual weights to provide granular partial credit
• Test execution ran in the isolated Docker container in which the agent completed its work
• Weighted scoring: passed tests contributed their weight to the final test score

LLM-as-a-Judge (35% weight)

• Used Claude-4-Sonnet as an external judge to evaluate agent behavior
• Evaluated four primary components:
  • Action Output Success (35%): Valid XML actions, successful parsing, error recovery
  • Todo Usage & Planning (25%): Initial planning, task tracking, continuous updates
  • Phase Adherence (25%): Following the 5-phase workflow (Planning → Exploration → Refinement → Execution → Verification)
  • Tool Usage Effectiveness (15%): Appropriate tool selection, purposeful actions
• Applied quality modifiers for error recovery, discovery quality, and efficiency
• Penalized overthinking without action, gaming behaviors, and phase violations
• Scored on HOW the agent worked, not WHETHER the task was completed

Judge Evaluation System

To ensure the LLM judge provided accurate and consistent scoring during RL training, I developed a simple evaluation system:

• Created test cases showing different agent trajectories
• Tested multiple LLM models as judges including: Kimi K2, Qwen-3-Coder, Claude Sonnet 4, Claude Haiku 3.5, to compare scoring accuracy.
• Found Claude Sonnet 4 provided the most consistent and accurate scoring, correctly identifying issues like lack of exploration and overthinking
  • Unfortunately Sonnet-4 is extremely expensive, so it is not very affordable for a 32 rollout, 1650 step run! But it was the only model which understood a good from bad trajectory well enough.
  • Many other models (including Haiku 3.5) gave inflated scores to problematic agent behaviors, with some scoring 0.85-0.95 for agents that skipped critical phases

To analyze judge model performance:

# Run evaluation on a specific model
uv run python evaluation/llm_as_a_judge_evals/judge_eval.py --model openrouter/openai/gpt-4.1 --attempts 3

# Generate performance report showing best models
uv run python evaluation/llm_as_a_judge_evals/report.py

Top 5 Judge Models Performance:

Rank	Model	Pass Rate	Avg Score
1	Claude Sonnet 4	46.67%	0.26
2	Claude 3.5 Haiku	46.67%	0.70
3	Qwen3 Coder	26.67%	0.76
4	Devstral Medium	23.33%	0.50
5	Kimi K2	23.33%	0.53

Claude Sonnet 4 ranks #1 despite having the same pass rate as Haiku because its significantly lower average score (0.26 vs 0.70) indicates stricter, more accurate judging (on the eval dataset). Lower scores mean the model better identifies problematic agent behaviors that other judges miss.

Other models tested include: GPT-4.1, Gemma-3-27B-IT, Qwen3-32B, and Qwen3-235B-A22B.

Dynamic LLM Judge Switching

To handle overloaded models, token limits or performance requirements during long training runs, the infrastructure supports hot-swapping between different LLM judge backends:

• Runtime switching without interrupting training process
• Switch between Claude Code CLI and LiteLLM backends as needed
• Useful when hitting API token limits or budget constraints
• See switch_judge_backend.py and switching documentation

Example workflow:

# Start with Claude Code CLI
python training_scripts/launch_training.py prod_32b_8_gpus

# Need to change? Switch to Lite