Slow_Thinking_with_LLMs

We are STILL exploring the uncharted territory of o1-like reasoning systems.

Content List

• LLM Exploration Mechanism
• Code-Integrated Reasoning
• ICPC-Eval
• SimpleDeepSearcher
• OlymMATH
• R1-Searcher
• STILL-3
  • STILL-3-Tool-32B
  • STILL-3-1.5B-preview
• Virgo
• STILL-Hallucination Mitigation
• STILL-2
• STILL-1

News

• [12 August 2025] LLM Exploration Mechanism: We systematically analyze LLM exploration mechanisms in RLVR, quantifying differences between SFT and RL, revealing entropy-performance dynamics, and proposing strategies to enhance exploration and training efficiency. We open-source our code in project page
• [2 June 2025] Code-Integrated Reasoning: We propose an improved tool-augmented reinforcement learning approach for code-integrated reasoning that balances exploration and stability, achieving strong performance on five mathematical benchmarks. We further provide an in-depth analysis of the mechanisms behind code-integrated reasoning, explaining why and how it is effective. For More details, please refer to our Project Page and Model.
• [29 May 2025] ICPC-Eval: We introduce ICPC-Eval, a new benchmark of 118 ICPC problems for evaluating LLM reasoning in competitive coding, featuring realistic ICPC competition scenario, robust local evaluation, and a iterative repair metrics Refine@K. We open-source our dataset, evaluation code, and paper. For more details, please refer to our project page and huggingface .
• [11 April 2025] SimpleDeepSearcher: We propose the SimpleDeepSearcher framework, which aims to stimulate autonomous web search capabilities in large language models through knowledge distillation and self-distillation. By leveraging powerful reasoning models and a real-world web search environment, we carefully curated and filtered 871 high-quality training examples, significantly enhancing model performance on complex information retrieval tasks and outperforming existing reinforcement learning approaches. All models and efficient fine-tuning datasets (0.5k and 0.8k) have been open-sourced.For more details, please refer to our project page and huggingface .
• [8 April 2025] ⚡️⚡️We open source our code and training data of STILL-3!
• [28 Mar 2025] OlymMATH: We introduce OlymMATH, a challenging benchmark of 200 Olympiad-level math problems across algebra, geometry, number theory, and combinatorics in both English and Chinese. Even the most advanced models achieve only moderate accuracy on OlymMATH-EN-HARD, highlighting significant room for improvement in mathematical reasoning. We open-source our dataset, evaluation code, and paper. For more details, please refer to our project page and huggingface .
• [9 Mar 2025] R1-Searcher: We propose R1-Searcher, a novel two-stage outcome-based RL approach designed to enhance the search capabilities of LLMs.
  • This method allows LLMs to autonomously invoke external search systems to access additional knowledge during the reasoning process.
  • Our framework relies exclusively on RL, without requiring process rewards or distillation for a cold start.
  • We conduct training on both base models (zero-shot) and fine-tuned models, analyzing key research questions arising during the training process.
  • **We open-source our models: Qwen-2.5-7B-Base-RL and Llama-3.1-8B-Instruct-RL, code and data.
• [7 Mar 2025] STILL-3: We propose STILL-3, An Empirical Study on Eliciting and Improving R1-like Reasoning Models.
  • We systematically experiment with and document the effects of various factors influencing RL training, conducting experiments on both base models (zero) and fine-tuned models.
  • Beyond RL training, we also explore the use of tool manipulation, finding that it significantly boosts the reasoning performance of large reasoning models.
  • We open source our code and training data!
• [1 Mar 2025] STILL-3-Tool-32B: We propose STILL-3-Tool-32B, leveraging python code to help the reasoning process. During evaluation, STILL-3-Tool-32B achieves 81.70% accuracy on AIME 2024, matching the performance of o3-mini, outperforming o1 and DeepSeek-R1. We open-source our code, model, and data. For more details, please refer to our Notion page.
• [26 Jan 2025] STILL-3-1.5B-preview: We release STILL-3-1.5B-preview, a 1.5B slow-thinking reasoning model achieves 39.33% accuracy on AIME benchmark! We utilize 30k queries to adapt reinforcement learning on 1.5B model (DeepSeek-R1-Distill-Qwen-1.5B) and observe the continuous performance improvement as the number of training steps increased. For better reproducing our work and advancing research progress, we open-source our code, model, and data.
• [6 Jan 2025] Virgo: We develop Virgo, a multi-modal slow-thinking reasoning model, based on Qwen2-VL-72B-Instruct, which achieves leading performance on four challenging multi-modal benchmarks. We demonstrate that the slow-thinking reasoning ability can be transferred from text to vision. We open-source the model and training data.
• [3 Jan 2025] STILL-Hallucination Mitigation: We propose HaluSearch, a framework that integrates tree search algorithms and a dynamic system switch mechanism, inspired by dual process theory, to reduce LLM hallucinations during inference.
• [22 Dec 2024] We open-source part of the training data in Github or HuggingFace and the model for community researchers to use for research purposes.
• [12 Dec 2024] STILL-2: We preliminarily reproduce a slow-thinking reasoning system, achieving competitive performance compared to industry-level reasoning systems on these benchmarks! And we also release the technical report, which presents the details about our reproduction.
• [18 Nov 2024] STILL-1: We release our first technical report, where we leverage reward-guided tree search algorithm to assist LLM reasoning process and largely enhance the performance of LLM on complex reasoning tasks.

Detailed Contents

From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR [Report]

We conduct a comprehensive study on LLM exploration mechanisms in RLVR, aiming to quantify, characterize, and optimize how large language models explore solution spaces during complex problem-solving.

• Core Focus & Analysis: We conduct a systematic analysis of the exploration capabilities of large language models (LLMs), examining key aspects such as the quantification of exploration boundaries, the interplay between entropy and performance, and strategies for enhancing data efficiency in reinforcement learning. Additionally, we compare the exploration behaviors of supervised fine-tuning (SFT) models, reinforcement learning (RL) models, and tool-integrated configurations, shedding light on their respective impacts on the breadth and efficiency of exploration.

• Key Findings:

• SFT expands exploration boundaries, while RL sharpens single-attempt accuracy but constricts exploration scope.

• entropy-performance during RL dynamics shift with training stages (rising stage curbs errors to form reasoning patterns, plateau stage refines high-entropy key tokens), instance quality (low-PPL samples enable more robust learning), and token positions (later tokens exert greater influence