mlx-vlm

MLX-VLM

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) and Omni Models (VLMs with audio and video support) on your Mac using MLX.

Table of Contents

• Installation
• Usage
  • Command Line Interface (CLI)
    • Thinking Budget
  • Speculative Decoding
    • DFlash, DFlash2, and DSpark
    • Gemma 4 MTP
    • Gemma 4 EAGLE-3
    • MiniMax M3 EAGLE-3
  • Chat UI with Gradio
  • Python Script
  • Server (FastAPI)
    • Continuous Batching
    • Automatic Prefix Caching (APC)
    • KV Cache Quantization
• 1-bit Affine Inference
• Activation Quantization (CUDA)
• Multi-Image Chat Support
  • Supported Models
  • Usage Examples
• Model-Specific Documentation
• Vision Feature Caching
• TurboQuant KV Cache
• Distributed Inference
• Fine-tuning

Model-Specific Documentation

Some models have detailed documentation with prompt formats, examples, and best practices:

Model Documentation DeepSeek-OCR Docs DeepSeek-OCR-2 Docs Unlimited-OCR Docs DOTS-OCR Docs DOTS-MOCR Docs ERNIE 4.5 VL Docs GLM-OCR Docs Phi-4 Reasoning Vision Docs MiniCPM-o Docs PaddleOCR-VL Docs Phi-4 Multimodal Docs MolmoPoint Docs LocateAnything Docs Moondream2 Docs Moondream3 Docs Gemma 4 Docs MiniMax M3 Docs Falcon-OCR Docs IndicOCR Docs PP-DocLayoutV3 Docs Granite Vision 3.2 Docs Granite 4.0 Vision Docs MiniCPM-V 4.6 Docs GLiNER2.5 Docs LLaVA-OneVision Docs K2-Horizon Docs Z1T-0 Docs Spark-X2.5 Docs

Installation

The easiest way to get started is to install the mlx-vlm package using pip:

pip install -U mlx-vlm

The Gradio chat UI needs an extra dependency that is not part of the base install:

pip install -U 'mlx-vlm[ui]'

Quote the package name so that shells which expand square brackets, such as zsh, do not treat [ui] as a glob pattern.

Agent Skills

This repo ships an agent-skills bundle under skills/ for common MLX-VLM workflows — usage, conversion, development, and support. Skills load into a coding agent (Claude Code, Codex, Gemini) so it follows the right project conventions instead of guessing.

Skill Description cli-inference Run and debug command-line inference (mlx_vlm.generate) — text/image/audio inputs and image-generation flags. server-inference Run and debug the local server across the models, chat, responses, messages, audio, image, cache, and metrics endpoints. convert-quantize Convert and quantize Hugging Face models to MLX (mlx_vlm.convert) — bits/group size, quant modes, RTN/AWQ, mixed recipes. add-new-model Port a new architecture into mlx_vlm/models — config, weight-name mapping, reuse a similar model, add a test class. benchmarking Produce credible, reproducible perf numbers and fork-vs-main A/B tables for PRs. contributing Shape a change to pass review — code/config/test placement, pre-commit hooks, and PR expectations. hf-cache-models List MLX-VLM-supported (and, with --check-arch, loadable) models in the local Hugging Face cache. reproducible-github-issues Turn CLI or server failures into concise, reproducible GitHub issues.

Validate the bundle at any time:

python3 skills/scripts/validate_skills.py

Install from a local checkout:

# Claude Code
/plugin marketplace add /path/to/mlx-vlm
/plugin install mlx-vlm-skills@mlx-vlm

# Codex CLI
codex plugin marketplace add /path/to/mlx-vlm
codex plugin add mlx-vlm-skills@mlx-vlm

# Gemini CLI
gemini extensions install /path/to/mlx-vlm/skills

Usage

Command Line Interface (CLI)

Generate output from a model using the CLI:

…

Thinking Budget

For thinking models (e.g., Qwen3.5), you can limit the number of tokens spent in the thinking block:

mlx_vlm.generate --model mlx-community/Qwen3.5-2B-4bit \
  --thinking-budget 50 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>" \
  --enable-thinking \
  --prompt "Solve 2+2"

Flag Description --enable-thinking Activate thinking mode in the chat template --thinking-budget Max tokens allowed inside the thinking block --thinking-start-token Token that opens a thinking block (default: <think>) --thinking-end-token Token that closes a thinking block (default: </think>)

When the budget is exceeded, the model is forced to emit \n</think> and transition to the answer. If --enable-thinking is passed but the model's chat template does not support it, the budget is applied only if the model generates the start token on its own.

On the server, thinking mode is disabled by default. Start the server with --enable-thinking to make thinking mode the default for requests that do not specify it:

mlx_vlm.server --model Qwen/Qwen3.5-4B --enable-thinking

You can also set server defaults for the thinking budget and delimiter tokens:

mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --enable-thinking \
  --thinking-budget 512 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>"

Requests can override the server defaults with enable_thinking, thinking_budget, thinking_start_token, or thinking_end_token.

Speculative Decoding

Speed up generation by drafting several candidate tokens with a small "drafter" model and verifying them in a single target forward pass. Three drafter families are supported.

Flag Description --draft-model HuggingFace repo or local path for the drafter --draft-kind Drafter family — dflash (default), eagle3, or mtp (native/assistant MTP) --draft-block-size Override the drafter's configured block size

See docs/usage.md for Python API examples including batch generation.

DFlash, DFlash2, and DSpark

A lightweight block-diffusion drafter that predicts multiple tokens per round, typically 2–3× faster.

# Text generation with speculative decoding
mlx_vlm.generate --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 --temperature 0 --enable-thinking

# Also works with images
mlx_vlm.generate --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash \
  --image examples/images/cats.jpg \
  --prompt "Describe this image." \
  --max-tokens 256 --temperature 0 --enable-thinking

# Server with speculative decoding
mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash

DFlash2 adds dynamic convolutions and a candidate-path selector. The published Qwen3.8-27B checkpoint is auto-detected and uses the shared exact DFlash target verification path. For the fastest quantized setup, convert the drafter to 4-bit; the verifier adapts between three and five rows from recent acceptance:

mlx_vlm.convert --hf-path z-lab/Qwen3.8-27B-DFlash2 \
  --mlx-path Qwen3.8-27B-DFlash2-4bit \
  --quantize --q-bits 4 --q-group-size 64

mlx_vlm.generate --model mlx-community/Qwen3.8-27B-4bit \
  --draft-model Qwen3.8-27B-DFlash2-4bit \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 --temperature 0

mlx_vlm.server --model mlx-community/Qwen3.8-27B-4bit \
  --draft-model Qwen3.8-27B-DFlash2-4bit

Liquid AI's DSpark checkpoint uses a Qwen3-style block drafter plus a learned Markov correction head. It is auto-detected and runs through the exact target verification path:

mlx_vlm.generate --model LiquidAI/LFM2.5-2.6B \
  --draft-model LiquidAI/LFM2.5-2.6B-DSpark \
  --prompt "Explain speculative decoding in three sentences." \
  --max-tokens 256 --temperature 0

mlx_vlm.server --model LiquidAI/LFM2.5-2.6B \
  --draft-model LiquidAI/LFM2.5-2.6B-DSpark

The published DSpark block_size: 9 means nine proposals, or ten target rows after adding the anchor token. On MLX, DSpark verifies seven proposals plus the anchor by default: eight rows exactly fill the verifier threadgroup, while nine or ten rows pad to sixteen and run slower. The trained width remains available with --draft-block-size 10. The checkpoint's confidence head is loaded for parity, and DSpark decoding currently requires greedy sampling (temperature=0).

Muse Glimmer's published assistant checkpoint is auto-detected as DFlash:

mlx_vlm.generate --model meta-models/Muse-Glimmer-30B \
  --draft-model meta-models/Muse-Glimmer-30B-assistant \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 --temperature 0

mlx_vlm.server --model meta-models/Muse-Glimmer-30B \
  --draft-model meta-models/Muse-Glimmer-30B-assistant

DFlash draft-cache windowing is available from the Python API. During speculative decoding the target model still verifies every proposed token with its full KV cache; this knob only changes the DFlash drafter cache. Wh