minions

Where On-Device and Cloud LLMs Meet

What is this? Minions is a communication protocol that enables small on-device models to collaborate with frontier models in the cloud. By only reading long contexts locally, we can reduce cloud costs with minimal or no quality degradation. This repository provides a demonstration of the protocol. Get started below or see our paper and blogpost below for more information.

Paper: Minions: Cost-efficient Collaboration Between On-device and Cloud Language Models

Minions Blogpost: https://hazyresearch.stanford.edu/blog/2025-02-24-minions

Secure Minions Chat Blogpost: https://hazyresearch.stanford.edu/blog/2025-05-12-security

Table of Contents

Looking for Secure Minions Chat? If you're interested in our end-to-end encrypted and chat system, please see the Secure Minions Chat README for detailed setup and usage instructions.

• Setup
  • Step 1: Clone and Install
  • Step 2: Install a Local Model Server
  • Step 3: Set Cloud LLM API Keys
• Minions Demo Application
• Minions WebGPU App
• Example Code
  • Minion (Singular)
  • Minions (Plural)
• Python Notebook
• Docker Support
• Command Line Interface
• Secure Minions Local-Remote Protocol
• Secure Minions Chat
• Apps
• Inference Estimator
  • Command Line Usage
  • Python API Usage
• Miscellaneous Setup
  • Using Azure OpenAI
• Maintainers

Setup

We have tested the following setup on Mac and Ubuntu with Python 3.10-3.11 (Note: Python 3.13 is not supported)

Optional: Create a virtual environment with your favorite package manager (e.g. conda, venv, uv)

conda create -n minions python=3.11

Step 1: Clone the repository and install the Python package.

git clone https://github.com/HazyResearch/minions.git
cd minions
pip install -e .  # installs the minions package in editable mode

note: for optional MLX-LM install the package with the following command:

pip install -e ".[mlx]"

note: for secure minions chat, install the package with the following command:

pip install -e ".[secure]"

note: for optional Cartesia-MLX install, pip install the basic package and then follow the instructions below.

Step 2: Install a server for running the local model.

We support three servers for running local models: lemonade, ollama, and tokasaurus. You need to install at least one of these.

• You should use ollama if you do not have access to NVIDIA/AMD GPUs. Install ollama following the instructions here. To enable Flash Attention, run launchctl setenv OLLAMA_FLASH_ATTENTION 1 and, if on a mac, restart the ollama app.
• You should use lemonade if you have access to local AMD CPUs/GPUs/NPUs. Install lemonade following the instructions here.
  • See the following for supported APU configurations: https://ryzenai.docs.amd.com/en/latest/llm/overview.html#supported-configurations
  • After installing lemonade make sure to launch the lemonade server. This can be done via the one-click Windows GUI installer which installs the Lemonade Server as a standalone tool.
  • Note: Lemonade does not support the Minion-CUA protocol at this time.
• You should use tokasaurus if you have access to NVIDIA GPUs and you are running the Minions protocol, which benefits from the high-throughput of tokasaurus. Install tokasaurus with the following command:

pip install tokasaurus

Optional: Install Cartesia-MLX (only available on Apple Silicon)

1. Download XCode
2. Install the command line tools by running xcode-select --install
3. Install the Nanobind

pip install nanobind@git+https://github.com/wjakob/nanobind.git@2f04eac452a6d9142dedb957701bdb20125561e4

1. Install the Cartesia Metal backend by running the following command:

pip install git+https://github.com/cartesia-ai/edge.git#subdirectory=cartesia-metal

1. Install the Cartesia-MLX package by running the following command:

pip install git+https://github.com/cartesia-ai/edge.git#subdirectory=cartesia-mlx

Optional: Install llama-cpp-python

Installation

First, install the llama-cpp-python package:

# CPU-only installation
pip install llama-cpp-python

# For Metal on Mac (Apple Silicon/Intel)
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python

# For CUDA on NVIDIA GPUs
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

# For OpenBLAS CPU optimizations
CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS" pip install llama-cpp-python

For more installation options, see the llama-cpp-python documentation.

Basic Usage

The client follows the basic pattern from the llama-cpp-python library:

from minions.clients import LlamaCppClient

# Initialize the client with a local model
client = LlamaCppClient(
    model_path="/path/to/model.gguf",
    chat_format="chatml",     # Most modern models use "chatml" format
    n_gpu_layers=35           # Set to 0 for CPU-only inference
)

# Run a chat completion
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What's the capital of France?"}
]

responses, usage, done_reasons = client.chat(messages)
print(responses[0])  # The generated response

Loading Models from Hugging Face

You can easily load models directly from Hugging Face:

client = LlamaCppClient(
    model_path="dummy",  # Will be replaced by downloaded model
    model_repo_id="TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
    model_file_pattern="*Q4_K_M.gguf",  # Optional - specify quantization
    chat_format="chatml",
    n_gpu_layers=35     # Offload 35 layers to GPU
)

Step 3: Set your API key for at least one of the following cloud LLM providers.

If needed, create an OpenAI API Key or TogetherAI API key or DeepSeek API key or MiniMax API key for the cloud model.

…

Minions Demo Application

To try the Minion or Minions protocol, run the following commands:

pip install torch transformers

streamlit run app.py

If you are seeing an error about the ollama client,

An error occurred: Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download

try running the following command:

OLLAMA_FLASH_ATTENTION=1 ollama serve

Minions WebGPU App

The Minions WebGPU app demonstrates the Minions protocol running entirely in the browser using WebGPU for local model inference and cloud APIs for supervision. This approach eliminates the need for local server setup while providing a user-friendly web interface.

Features

• Browser-based: Runs entirely in your web browser with no local server required
• WebGPU acceleration: Uses WebGPU for fast local model inference
• Model selection: Choose from multiple pre-optimized models from MLC AI
• Real-time progress: See model loading progress and conversation logs in real-time
• Privacy-focused: Your API key and data never leave your browser

Quick Start

1. Navigate to the WebGPU app directory:

   bash复制

   cd apps/minions-webgpu

2. Install dependencies:

   bash复制

   npm install

3. Start the development server:

   bash复制

   npm start

4. Open your browser and navigate to the URL shown in the terminal (typically http://localhost:5173)

Example code: Minion (singular)

The following example is for an ollama local client and an openai remote client. The protocol is minion.

…

Example Code: Minions (plural)

The following example is for an ollama local client and an openai remote client. The protocol is minions.

…

Python Notebook

To run Minion/Minions in a notebook, checkout minions.ipynb.

Docker support

Build the Docker Image

docker build -t minions-docker .

Run the Container

Note: The container automatically starts an Ollama service in the background for local model inference. This allows you to use models like llama3.2:3b without additional setup.

# Basic usage (includes Ollama service)
docker run -i minions-docker

# With Docker socket mounted (for Docker Model Runner)
docker run -i -v /var/run/docker.sock:/var/run/docker.sock minions-docker

# With API keys for remote models
docker run -i -e OPENAI_API_KEY=your_key -e ANTHROPIC_API_KEY=your_key minions-docker

# With custom Ollama host
docker run -i -e OLLAMA_HOST=0.0.0.0:11434 minions-docker

# For Streamlit app (legacy usage)
docker run -p 8501:8501 --env OPENAI_API_KEY= --env DEEPSEEK_API_KEY= minions-docker

Docker Minion Protocol Usage

The Docker container supports a stdin/stdout interface for running various minion protocols. It expects JSON input with the following structure:

{
  "local_client": {
    "type": "ollama",
    "model_name": "llama3.2:3b",
    "port": 11434,
    "kwargs": {}
  },
  "remote_client": {
    "type": "openai",
    "model_name": "gpt-4o",
    "kwargs": {
      "api_key": "your_openai_key"
    }
  },
  "protocol": {
    "type": "minion",
    "max_rounds": 3,
    "log_dir": "minion_logs",
    "kwargs": {}
  },
  "call_params": {
    "task": "Your task here",
    "context": ["Context string 1", "Context string 2"],
    "max_rounds": 2
  }
}

Usage Examples

Basic Minion Protocol:

echo '{
  "local_client": {
    "type": "ollama",
    "model_name": "llama3.2:3b"
  },
  "remote_client": {
    "type": "openai",
    "model_name": "gpt-4o"
  },
  "protocol": {
    "type": "minion",
    "max_rounds": 3
  },
  "call_params": {
    "task": "Analyze the patient data and provide a diagnosis",
    "context": ["Patient John Doe is a 60-year-old male with hypertension. Blood pressure: 160/100 mmHg. LDL cholesterol: 170 mg/dL."]
  }
}' | docker run -i -e OPENAI_API_KEY=$OPENAI_API_KEY minions-docker

Minions (Parallel) Protocol:

echo '{
  "local_client": {
    "type": "ollama",
    "model_name": "llama3.2:3b"
  },
  "remote_client": {
    "type": "openai",
    "model_name": "gpt-4o"
  },
  "protocol": {
    "type": "minions"
  },
  "call_params": {
    "task": "Analyze the financial data and extract key insights",
    "doc_metadata": "Financial report",
    "context": ["Revenue increased by 15% year-over-year. Operating expenses rose by 8%. Net profit margin improved to 12%."]
  }
}' | docker run -i -e OPENAI_API_KEY=$OPENAI_API_KEY minions-docker

Supported Client Types

Local Clients:

• ollama: Uses Ollama for local inference (included in container)
• docker_model_runner: Uses Docker Model Runner for local inference

Remote Clients:

• openai: OpenAI API
• anthropic: Anthropic API

Supported Protocol Types

• minion: Single conversation protocol
• minions: Parallel processing protocol

Output Format

The container outputs JSON with the following structure:

{
  "success": true,
  "result": {
    "final_answer": "The analysis result...",