atropos

⚠️ This repository has been archived and is no longer maintained.

No further updates, bug fixes, or security patches will be made.

Atropos - Nous Research's LLM RL Gym

What is Atropos?

Atropos is an environment microservice framework for async RL with LLMs.

Atropos encompasses both environments, which are set up as services, and a trajectory API for the environments to send data to and for the trainer to pull batches from.

Atropos is a robust, scalable framework for Reinforcement Learning Environments with LLMs.

The goal: provide a flexible, scalable, and standardized platform to accelerate LLM-based RL research across diverse, interactive settings.

The framework supports collecting, distributing and evaluating LLM trajectories through diverse environments including:

Experimental results from models trained using Atropos' environments

We have been able to achieve significant improvements on specific domains or tasks with Atropos - Below are some of the results.

Tool Calling Environment Results:

Model Artifact: https://huggingface.co/NousResearch/DeepHermes-ToolCalling-Specialist-Atropos

Environment Used: https://github.com/NousResearch/atropos/blob/main/environments/tool_calling_server.py

Financial Fundamentals Prediction Environment Results:

Model Artifact: https://huggingface.co/NousResearch/DeepHermes-Financial-Fundamentals-Prediction-Specialist-Atropos

Environment Used: https://github.com/NousResearch/atropos/blob/main/environments/fundamental_prediction_environment.py

RLAIF Experiment Artifacts

Using the RLAIF Environment to change the personality of the model, we have produced several artifacts of interesting and weird personalities.

DeepHermes Egregore v1 and v2 8B:

https://huggingface.co/NousResearch/DeepHermes-Egregore-v1-RLAIF-8b-Atropos https://huggingface.co/NousResearch/DeepHermes-Egregore-v2-RLAIF-8b-Atropos

DeepHermes Ascension Maze 8B:

https://huggingface.co/NousResearch/DeepHermes-AscensionMaze-RLAIF-8b-Atropos

Environment Used: https://github.com/NousResearch/atropos/blob/main/environments/rlaif_server.py

Navigating the Repo

Key Documents:

• Base Environment Class - Documentation for creating custom environments
• ManagedServer Guide - Recommended approach for automatic token and logprob tracking
• Environments Overview and Contribution Guide - Documentation for existing environments and how to contribute new ones.
• Full Environment Config Options - Documentation for creating custom environments
• Example Trainer - Getting started with training
• Slurm Guide - Guide for using Atropos with Slurm for distributed inference
• Frequently Asked Questions (FAQ) - Answers to common questions for new users
• Contributing Guide - Guidelines for contributors
• License - MIT license details

Prerequisites

Before installing Atropos, ensure you have the following:

• Python 3.10+ — Required. Check with python --version
• Git — For cloning the repository
• An OpenAI-compatible API endpoint — Atropos environments need an inference server. Options include:
  • A local vLLM or SGLang instance
  • An OpenAI API key (set as OPENAI_API_KEY environment variable)
  • Any provider with an OpenAI-compatible endpoint (e.g., Together AI, OpenRouter)
• Weights & Biases account (optional) — For experiment tracking. Set use_wandb=False in your environment config to skip

Note: You do not need a GPU to develop or test environments locally. A GPU is only required for running inference servers locally or for training.

Installation

Get your Python 3.10 (or later) environment ready, then simply pip install:

pip install atroposlib

If you're looking to get into developing the repo or using the environments:

pip install -e .                # for using
pip install -e .[dev]           # for development
pip install -e .[examples]      # for running examples
pip install -e .[verifiers]     # for verifiers integration
pip install -e .[all]           # for everything

Important: If you're committing to the repository, please install the pre-commit hooks:

pre-commit install

Quick Start Guide

1. Create Your First Environment

   • Review our Base Class Documentation to understand the core concepts
   • Check out existing environments in the environments/ directory for examples

2. Run an Example Environment

You should edit the config_init section of the environment file you want (For example, in GSM8K Environment) to point to a running VLLM or SGLang inference server as well as any other configuration changes you'd like to make, such as the group size, then:

Note: By default, Atropos uses the OpenAI-compatible API endpoint which works with any provider. For enhanced features, use VLLMServer (atroposlib/envs/server_handling/vllm_server.py) or SGLangServer (atroposlib/envs/server_handling/sglang_server.py) for direct access to native APIs with full token and logprob tracking.

# Start the API server
run-api

In a separate terminal, start the GSM8K environment microservice

python environments/gsm8k_server.py serve --openai.model_name Qwen/Qwen2.5-1.5B-Instruct --slurm false
# alternatively
# python environments/gsm8k_server.py serve --config environments/configs/example.yaml
# python environments/gsm8k_server.py serve --config environments/configs/example.yaml --env.group_size 8 # cli args override corresponding config settings

1. Grabbing Rollouts

If you want to just start getting rollouts, and not use a trainer, see the debug section for help getting started with the available tools, we recommend starting with process or view-run

1. Training Your Model
   • Follow our training example guide for detailed instructions
   • Monitor progress through our built-in logging and reporting system:
     • Completion lengths
     • Evaluation accuracies
     • Full rollouts and scores

You can use multiple environments at once, just point them all to the same server.

Environments come with detailed logging and reporting support, runs track completion lengths, eval accuracies, full rollouts and scores, and more:

Trainer Integrations

Axolotl

Axolotl is a powerful tool for fine-tuning a wide range of AI models, supporting techniques like LoRA and QLoRA through simple YAML configurations.

The Atropos plugin for Axolotl seamlessly integrates Atropos' RL environments into Axolotl's training pipelines. This allows you to leverage Atropos for reinforcement learning while utilizing Axolotl's extensive features for model fine-tuning.

To use, follow the README on the plugin repository.

Tinker

The Tinker API is a simple and flexible LoRA trainer framework for researchers and developers to quickly build out their ideas without worrying about the complexities of distributed training. Users write a simple loop that runs on their CPU, and Tinker manages the backend computation on their GPUs, while still providing full control over the training and algorithmic details.

The Tinker-Atropos integration layer enables all Atropos environments to leverage the power of Tinker for their RL experiments. This allows users with little or no compute to develop and build Atropos environments with minimal worry about the underlying compute behavior, as well as providing an easy environment integration point for Tinker users.

To get started, check out the README at the project repository.

Atropos' Example Trainer

Atropos repo contains an example trainer that should primarily be used as a reference example to show how a trainer and inference provider can be integrated with Atropos to complete the RL Training Loop.

To use the example trainer, see this page: training example guide

On-Policy Distillation (API + ScoredDataGroup Contract)

Atropos now supports OPD at the transport layer by carrying distillation arrays through ScoredDataGroup and the API queue/batch endpoints.

Scope of this change

• No teacher fetching/orchestration in BaseEnv.
• Environments or external pipelines are responsible for populating distillation arrays.
• API stores and returns those arrays unchanged.

Distillation payload fields

Each scored group may include:

• distill_token_ids: shape [sequence][position][top_k]
• distill_logprobs: shape [sequence][position][top_k]

These fields are optional, and when present are forwarded from:

• environment -> /scored_data or /scored_data_list
• API queue -> /batch -> trainer

Minimal producer example (environment side)

scores["distill_token_ids"] = distill_token_ids
scores["distill_logprobs"] = distill_logprobs

Minimal consumer check (trainer/debug side)

curl -s http://localhost:8002/latest_example | jq '{has_ids:(.distill_token_ids!=null), has_lps:(.distill_logprobs!=null)}'

Notes

• The API does not validate cross-field semantics beyond schema typing.
• Trainers should validate alignment assumptions they require (sequence length, per-position top-k, etc.).
• Teacher-side architecture and prompt/rendering strategy are intentionally out of scope for this PR.

TeacherDistillationEnv follow-up

The follow-up teacher environment uses a dedicated teacher server config and attaches teacher prompt logprobs before the group is sent to the API.

Teacher config shape:

TeacherDistillationConfig(
    teacher_enabled=True,
    teacher_top_k=8,
)

Teacher server configs are passed separately at init, just like the primary server_configs:

env = MyTeacherEnv(
    config=env_config,
    server_configs=student_server_configs,
    teacher_server_configs=[
        APIServerConfig(
            base_url="http://localhost:9003/v1",
            model_name="Qwen/Qwen3-30B-A3B-Instruct-2507",
            api_key="",
            server_type="vllm",
            tokenizer_name="Qwen/Qwen3-30B-A3B-Instruct-2507",
        )
    ],
)

You can either:

• build a teacher-enabled env by mixing TeacherDistillationEnv into an existing BaseEnv-derived env such as GSM8kEnv, or
• subclass TeacherDistillationEnv directly and implement the usual environment methods yourself.

In both cases, TeacherDistillationEnv still assumes the normal BaseEnv runtime contract: tokenized rollouts, ScoredDataGroup pay