Awesome-System2-Reasoning-LLM

Awesome-System2-Reasoning-LLM

Updates

• 2025.02: We released a survey paper "From System 1 to System 2: A Survey of Reasoning Large Language Models". Feel free to cite or open pull requests.

Introduction

Welcome to the repository for our survey paper, "From System 1 to System 2: A Survey of Reasoning Large Language Models". This repository provides resources and updates related to our research. For a detailed introduction, please refer to our survey paper.

Achieving human-level intelligence requires enhancing the transition from System 1 (fast, intuitive) to System 2 (slow, deliberate) reasoning. While foundational Large Language Models (LLMs) have made significant strides, they still fall short of human-like reasoning in complex tasks. Recent reasoning LLMs, like OpenAI’s o1, have demonstrated expert-level performance in domains such as mathematics and coding, resembling System 2 thinking. This survey explores the development of reasoning LLMs, their foundational technologies, benchmarks, and future directions. We maintain an up-to-date GitHub repository to track the latest developments in this rapidly evolving field.

This image highlights the progression of AI systems, emphasizing the shift from rapid, intuitive approaches to deliberate, reasoning-driven models. It shows how AI has evolved to handle a broader range of real-world challenges.

The recent timeline of reasoning LLMs, covering core methods and the release of open-source and closed-source reproduction projects.

Table of Contents

• Awesome-System-2-AI
  • Part 1: O1 Replication
  • Part 2: Process Reward Models
  • Part 3: Reinforcement Learning
  • Part 4: MCTS/Tree Search
  • Part 5: Self-Training / Self-Improve
  • Part 6: Reflection
  • Part 7: Efficient System2
  • Part 8: Explainability
  • Part 9: Multimodal Agent related Slow-Fast System
  • Part 10: Benchmark and Datasets
  • Part 11: Reasoning and Safety
  • Part 12: R1 Driven Multimodal Reasoning Enhancement

Part 1: O1 Replication

• O1 Replication Journey: A Strategic Progress Report -- Part 1 [Paper]
• Enhancing LLM Reasoning with Reward-guided Tree Search [Paper]
• Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions [Paper]
• O1 Replication Journey--Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson? [Paper]
• Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems [Paper]
• o1-Coder: an o1 Replication for Coding [Paper]
• HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs [Paper]
• DRT: Deep Reasoning Translation via Long Chain-of-Thought [Paper]
• mini-deepseek-r1 [Blog]
• Run DeepSeek R1 Dynamic 1.58-bit [Blog]
• Simple Reinforcement Learning for Reasoning [Notion]
• TinyZero [github]
• Open R1 [github]
• Search-o1: Agentic Search-Enhanced Large Reasoning Models [Paper]
• Virgo: A Preliminary Exploration on Reproducing o1-like MLLM [Paper]
• The Relationship Between Reasoning and Performance in Large Language Models -- o3 (mini) Thinks Harder, Not Longer [Paper]
• Open-Reasoner-Zero [Paper]
• X-R1 [github]
• Unlock-Deepseek [Blog]
• Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning [Paper]
• LLM-R1 [github]

Part 2: Process Reward Models

• Solving Math Word Problems with Process and Outcome-Based Feedback [Paper]
• Improve Mathematical Reasoning in Language Models by Automated Process Supervision [Paper]
• Making Large Language Models Better Reasoners with Step-Aware Verifier [Paper]
• Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations [Paper]
• OVM: Outcome-supervised Value Models for Planning in Mathematical Reasoning [Paper]
• Let's Verify Step by Step. [Paper]
• Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs [Paper]
• AutoPSV: Automated Process-Supervised Verifier [Paper]
• ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search [Paper]
• Free Process Rewards without Process Labels. [Paper]
• Outcome-Refining Process Supervision for Code Generation [Paper]
• PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models. [Paper]
• ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding [Paper]
• The Lessons of Developing Process Reward Models in Mathematical Reasoning. [Paper]
• ToolComp: A Multi-Tool Reasoning & Process Supervision Benchmark. [Paper]
• ARMAP: Scaling Autonomous Agents via Automatic Reward Modeling And Planning [Paper]
• Uncertainty-Aware Step-wise Verification with Generative Reward Models [Paper]
• AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidence [Paper]
• Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models [Paper]
• Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling [Paper]
• Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems [Paper]
• Unified Reward Model for Multimodal Understanding and Generation [Paper]
• Reward Shaping to Mitigate Reward Hacking in RLHF [Paper]
• Multi-head Reward Aggregation Guided by Entropy [Paper]
• [Paper]
• Better Process Supervision with Bi-directional Rewarding Signals [Paper]
• Inference-Time Scaling for Generalist Reward Modeling [Paper]

Part 3: Reinforcement Learning

• Improve Vision Language Model Chain-of-thought Reasoning [Paper]
• Does RLHF Scale? Exploring the Impacts From Data, Model, and Method [Paper]
• Offline Reinforcement Learning for LLM Multi-Step Reasoning [Paper]
• ReFT: Representation Finetuning for Language Models [Paper]
• InfAlign: Inference-aware language model alignment [Paper]
• Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling [Paper]
• Challenges in Ensuring AI Safety in DeepSeek-R1 Models: The Shortcomings of Reinforcement Learning Strategies [Paper]
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning [Paper]
• Kimi k1.5: Scaling Reinforcement Learning with LLMs [Paper]
• Deepseekmath: Pushing the limits of mathematical reasoning in open language models [Paper]
• Reasoning with Reinforced Functional Token Tuning [Paper]
• Value-Based Deep RL Scales Predictably [Paper]
• MM-RLHF: The Next Step Forward in Multimodal LLM Alignment [Paper]
• Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search [Paper]
• DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL [Paper]
• LIMR: Less is More for RL Scaling [Paper]
• A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematics [Paper]
• Med-RLVR: Emerging Medical Reasoning from a 3B base model via reinforcement Learning [Paper]
• QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search [Paper]
• Process Reinforcement through Implicit Rewards [Paper]
• UI-R1: Enhancing Action Prediction of GUI Agents by Reinforcement Learning [Paper]
• All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning [Paper]
• R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model [Paper]
• Visual-RFT: Visual Reinforcement Fine-Tuning [Paper]
• GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based VLM Agent Training [Paper]
• L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning [Paper]
• Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't [Paper]
• Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement [Paper]
• VLAA-Thinker [github]
• Concise Reasoning via Reinforcement Learning [Paper]
• d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning [github]
• Algorithm Discovery With LLMs: Evolutionary Search Meets Reinforcement Learning [Paper]
• Efficient Reinforcement Finetuning via Adaptive Curriculum Learning [Paper]
• VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning [Paper]
• SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Langu