LlamaFactory

Used by Amazon, NVIDIA, Aliyun, etc.

---

https://github.com/user-attachments/assets/9b7033e8-f08a-4c3f-bd33-547896664e6e

---

👋 Join our WeChat and NPU user groups.

[ English | 中文 ]

Fine-tuning a large language model can be easy as...

https://github.com/user-attachments/assets/3991a3a8-4276-4d30-9cab-4cb0c4b9b99e

Start local training:

• Please refer to usage

Start cloud training:

• Colab (free): https://colab.research.google.com/drive/1eRTPn37ltBbYsISy9Aw2NuI2Aq5CQrD9?usp=sharing
• PAI-DSW (free trial): https://gallery.pai-ml.com/#/preview/deepLearning/nlp/llama_factory
• AMD GPU Cloud (free credits): https://github.com/AMD-AIM/AMD_Developers_Notebooks/blob/main/en/AMD_developer_LLaMAFactory_note_en.md

Read technical notes:

• Documentation (WIP): https://llamafactory.readthedocs.io/en/latest/
• Documentation (AMD GPU): https://rocm.docs.amd.com/projects/ai-developer-hub/en/latest/notebooks/fine_tune/llama_factory_llama3.html
• Documentation (ASCEND NPU): https://llamafactory.readthedocs.io/en/latest/multibackend/npu/index.html
• Official Blog: https://blog.llamafactory.net/en/

[!NOTE] Except for the above links, all other websites are unauthorized third-party websites. Please carefully use them.

Table of Contents

• Features
• Blogs
• Changelog
• Supported Models
• Supported Training Approaches
• Provided Datasets
• Requirement
• Getting Started
  • Installation
  • Data Preparation
  • Quickstart
  • Fine-Tuning with LLaMA Board GUI
  • Build Docker
  • Deploy with OpenAI-style API and vLLM
  • Download from ModelScope Hub
  • Download from Modelers Hub
  • Use W&B Logger
  • Use SwanLab Logger
• Projects using LlamaFactory
• License
• Citation
• Acknowledgement

Features

• Various models: LLaMA, LLaVA, Mistral, Mixtral-MoE, Qwen3, Qwen3-VL, DeepSeek, Gemma, GLM, Phi, etc.
• Integrated methods: (Continuous) pre-training, (multimodal) supervised fine-tuning, reward modeling, PPO, DPO, KTO, ORPO, etc.
• Scalable resources: 16-bit full-tuning, freeze-tuning, LoRA and 2/3/4/5/6/8-bit QLoRA via AQLM/AWQ/GPTQ/LLM.int8/HQQ/EETQ.
• Advanced algorithms: GaLore, BAdam, APOLLO, Adam-mini, Muon, OFT, DoRA, LongLoRA, LLaMA Pro, Mixture-of-Depths, LoRA+, LoftQ and PiSSA.
• Practical tricks: FlashAttention-2, Unsloth, Liger Kernel, KTransformers, RoPE scaling, NEFTune and rsLoRA.
• Wide tasks: Multi-turn dialogue, tool using, image understanding, visual grounding, video recognition, audio understanding, etc.
• Experiment monitors: LlamaBoard, TensorBoard, Wandb, MLflow, SwanLab, etc.
• Faster inference: OpenAI-style API, Gradio UI and CLI with vLLM worker or SGLang worker.

Day-N Support for Fine-Tuning Cutting-Edge Models

Support Date	Model Name
Day 0	Qwen3 / Qwen2.5-VL / Gemma 3 / GLM-4.1V / InternLM 3 / MiniCPM-o-2.6
Day 1	Llama 3 / GLM-4 / Mistral Small / PaliGemma2 / Llama 4

Blogs

[!TIP] Now we have a dedicated blog for LlamaFactory!

Website: https://blog.llamafactory.net/en/

• 💡 KTransformers Fine-Tuning × LlamaFactory: Fine-tuning 1000 Billion models with 2 4090-GPU + CPU (English)
• 💡 Easy Dataset × LlamaFactory: Enabling LLMs to Efficiently Learn Domain Knowledge (English)
• 💡 DataFlow × LlamaFactory: Producing High-Quality Data for LLM Training with a Data Preparation Pipeline (English) | 中文
• 💡 DataFlex × LlamaFactory: A Data-Centric Dynamic Training System Built on LlamaFactory (English) | 中文
• A One-Stop Code-Free Model Reinforcement Learning and Deployment Platform based on LlamaFactory and EasyR1 (Chinese)
• How Apoidea Group enhances visual information extraction from banking documents with multimodal models using LlamaFactory on Amazon SageMaker HyperPod (English)

All Blogs

• LlamaFactory: Fine-tuning the DeepSeek-R1-Distill-Qwen-7B Model for News Classifier (Chinese)
• A One-Stop Code-Free Model Fine-Tuning & Deployment Platform based on SageMaker and LlamaFactory (Chinese)
• LlamaFactory Multi-Modal Fine-Tuning Practice: Fine-Tuning Qwen2-VL for Personal Tourist Guide (Chinese)
• LlamaFactory: Fine-tuning Llama3 for Role-Playing (Chinese)

Changelog

[25/10/26] We support Megatron-core training backend with mcore_adapter. See PR #9237 to get started.

[25/08/22] We supported OFT and OFTv2. See examples for usage.

[25/08/20] We supported fine-tuning the Intern-S1-mini models. See PR #8976 to get started.

[25/08/06] We supported fine-tuning the GPT-OSS models. See PR #8826 to get started.

Full Changelog

[25/07/02] We supported fine-tuning the GLM-4.1V-9B-Thinking model.

[25/04/28] We supported fine-tuning the Qwen3 model family.

[25/04/21] We supported the Muon optimizer. See examples for usage. Thank @tianshijing's PR.

[25/04/16] We supported fine-tuning the InternVL3 model. See PR #7258 to get started.

[25/04/14] We supported fine-tuning the GLM-Z1 and Kimi-VL models.

[25/04/06] We supported fine-tuning the Llama 4 model. See PR #7611 to get started.

[25/03/31] We supported fine-tuning the Qwen2.5 Omni model. See PR #7537 to get started.

[25/03/15] We supported SGLang as inference backend. Try infer_backend: sglang to accelerate inference.

[25/03/12] We supported fine-tuning the Gemma 3 model.

[25/02/24] Announcing EasyR1, an efficient, scalable and multi-modality RL training framework for efficient GRPO training.

[25/02/11] We supported saving the Ollama modelfile when exporting the model checkpoints. See examples for usage.

[25/02/05] We supported fine-tuning the Qwen2-Audio and MiniCPM-o-2.6 on audio understanding tasks.

[25/01/31] We supported fine-tuning the DeepSeek-R1 and Qwen2.5-VL models.

[25/01/15] We supported APOLLO optimizer. See examples for usage.

[25/01/14] We supported fine-tuning the MiniCPM-o-2.6 and MiniCPM-V-2.6 models. Thank @BUAADreamer's PR.

[25/01/14] We supported fine-tuning the InternLM 3 models. Thank @hhaAndroid's PR.

[25/01/10] We supported fine-tuning the Phi-4 model.

[24/12/21] We supported using SwanLab for experiment tracking and visualization. See this section for details.

[24/11/27] We supported fine-tuning the Skywork-o1 model and the OpenO1 dataset.

[24/10/09] We supported downloading pre-trained models and datasets from the Modelers Hub. See this tutorial for usage.

[24/09/19] We supported fine-tuning the Qwen2.5 models.

[24/08/30] We supported fine-tuning the Qwen2-VL models. Thank @simonJJJ's PR.

[24/08/27] We supported Liger Kernel. Try enable_liger_kernel: true for efficient training.

[24/08/09] We supported Adam-mini optimizer. See examples for usage. Thank @relic-yuexi's PR.

[24/07/04] We supported contamination-free packed training. Use neat_packing: true t