Tutel MoE: 优化的专家混合库,支持使用 FP8/NVFP4/MXFP4 的 GptOss/DeepSeek/Kimi-K2/Qwen3
Tutel MoE: 优化的专家混合库,支持使用 FP8/NVFP4/MXFP4 的 GptOss/DeepSeek/Kimi-K2/Qwen3
Tutel MoE: An Optimized Mixture-of-Experts Implementation, also the first parallel solution proposing "No-penalty Parallism/Sparsity/Capacity/.. Switching" for modern training and inference that have dynamic behaviors. Support direct NVFP4/MXFP4/BlockwiseFP8 Inference for MoE-based GLM-5.x / DeepSeek-3.x / Kimi-2.x / Kimi-3.x / Qwen3 / Gpt-OSS using A100/A800/H100/MI300/..
[!TIP]
Steps for Kimi-K3/GLM-5.x (Claude-Code Mode):
✔️ MI300x8/MI325x8 (192GB PCIe5) for Kimi K3 (2.8TB): ✔️ max-context-size = 1M, ✔️ Vision
✔️ MI300x1/MI325x1 (192GB PCIe5) for GLM-5.3-Flash (0.3TB): ✔️ max-context-size = 200K, ✔️ Vision
✔️ MI300x8/MI325x8 (192GB PCIe5) for GLM-5.x (0.8TB): ✔️ max-context-size = 1M, ❌ Vision
✔️ A100x8/H100x8 (80G SXM) for GLM-5.x (0.8TB): ✔️ max-context-size = 1M, ❌ Vision
Azure GPU Type vLLM/SGL Tutel AMD MI300X + GLM-5.3 (750W x8) 0 t/s (MTP=0, OoM) 761 t/s (MTP=0, GPU=8, BSZ=32) AMD MI300X + GLM-5.3-Flash (750W x1) 0 t/s (MTP=0, OoM) 108 t/s (MTP=0, GPU=1) 0 t/s (MTP=4, OoM) 225 t/s ×TAR (MTP=4, GPU=1) AMD MI300X + Kimi K3 (750W x8) 0 t/s (MTP=0, OoM) 73.4 t/s (MTP=0, GPU=8) 0 t/s (MTP=8, OoM) 287.1 t/s ×TAR (MTP=8, GPU=8) AMD MI325X + Kimi K3 (1000W x8) 3.1 t/s (MTP=0) 82.0 t/s (MTP=0, GPU=8) 0 t/s (MTP=8, OoM) 315.2 t/s ×TAR (MTP=8, GPU=8) AMD MI355X + Kimi K3 (1400W x8) 43.5 t/s (MTP=0) (TBD, no environment available) NVIDIA B200 + Kimi K3 (1000W x8) 0 t/s (MTP=0) (TBD, no environment available) Azure Service:
…Agent Examples:
# Vision Example in the container (for GLM-5.3-Flash and Kimi-K3 only): node-container:$ claude-unattended "What is this - https://i0.hdslb.com/bfs/archive/14c094d92aabe3f9a5c4356a66a122fc335239f2.jpg" # Text Example in the container: node-container:$ claude-unattended "What time is it in PST?"Agent Setup for Linux / WSL (Ubuntu >= 24.04):
…Agent Setup for Windows (>= 10.0):
…[!TIP]
Steps for Kimi-K2.6/2.7/DeepSeek V3.2 (Long-Context Mode):
…[!TIP]
Steps for Microsoft VibeVoice (Multimodality Mode):
…
Model & Machine Type Precision SGL (no-MTP) Tutel (no-MTP) $deepseek-ai/DeepSeek-V3.2\ (671B,\ A100 \times 8)$ nvfp4 - 102 $deepseek-ai/DeepSeek-V3.2\ (671B,\ MI300 \times 8)$ nvfp4 - 151 $moonshotai/Kimi-K2-Instruct\ (1T,\ A100 \times 8)$ nvfp4 - 104 $moonshotai/Kimi-K2-Instruct\ (1T,\ MI300 \times 8)$ fp8b128 49 153 $NVFP4/Qwen3-235B-A22B-Instruct-2507-FP4(A100\times8)$ nvfp4 - 114 $NVFP4/Qwen3-235B-A22B-Instruct-2507-FP4(MI300\times8)$ nvfp4 - 122 $openai/gpt-oss-120b\ (120B,\ A100 \times 1)$ mxfp4 127 212 $openai/gpt-oss-120b\ (120B,\ MI300 \times 1)$ mxfp4 191 311 $microsoft/VibeVoice-1.5B (A100 \times 1)$ bf16 - rtf=0.07 $microsoft/VibeVoice-1.5B (MI300 \times 1)$ bf16 - rtf=0.06
Image-20260902: Add Vision Support for Kimi-K3/GLM-5.3-Flash for MI300.
Image-20260831: Add GLM-5.3/GLM-5.3-Flash Support on MI300 192GB PCIe-5.
Image-20260825: Add DFlash Support for Kimi K3 on MI300 192GB PCIe-5.
Image-20260808: Support 1M context for Kimi K3 on MI300 192GB PCIe-5.
Image-20260707: Memory fixes for 1M context on A100 80GB SXM.
Image-20260618: Fit GLM-5.2 1M context into A100 80GB x 8 SXM.
Image-20260603: Improved Claude Code Tooling Performance for GLM-5/5.1.
Image-20260511: Initial Claude Code Support for GLM-5/5.1.
Image-20260327: Add support for Kimi-K2.5.
Image-20260306: Support DeepSeek V3.2 Long-context mode for A100/H100/MI300/B200.
Image-20251222: Fine-tune A100 performance for most models.
Image-20251111: Integrate Tutel LLM module into VibeVoice for accelerated inference (rtf = 0.07 for single A100).
Image-20251006: Resolve compatibility with DeepSeek-V3.2-Exp
Image-20250827: Add distributed support for OpenAI GPT-OSS 20B/120B with MXFP4 inference
Image-20250801: Support Qwen3 MoE series, integrate OpenWebUI
Image-20250712: Support Kimi K2 1TB MoE inference with NVFP4 for NVIDIA/AMD GPUs
Image-20250601: Improved decoding performance for DeepSeek 671B on MI300x to 140-150 TPS
More image versions can be found here
Tutel v0.4.2: Add R1-FP4/Qwen3MoE-FP8 Support for NVIDIA and AMD GPUs & Fast Gating APIs:
…Tutel v0.4.1: Support Deepseek R1 FP8 with NVIDIA GPUs (A100 / A800)
Tutel v0.4.0: Accelerating Deepseek R1 Full-precision-Chat for AMD MI300x8:
>> Example:
# Step-1: Download Deepseek R1 671B Model
huggingface-cli download deepseek-ai/DeepSeek-R1 --local-dir ./deepseek-ai/DeepSeek-R1
# Step-2: Using 8 MI300 GPUs to Serve Deepseek R1 Chat on Local Port :8000
docker run -it --rm --ipc=host --privileged -p 8000:8000 \
-v /:/host -w /host$(pwd) tutelgroup/deepseek-671b:mi300x8-chat-20250224 \
--model_path ./deepseek-ai/DeepSeek-R1
# Step-3: Issue a Prompt Request with curl
curl -X POST http://0.0.0.0:8000/chat -d '{"text": "Calculate the result of: 1 / (sqrt(5) - sqrt(3))"}'Tutel v0.3.3: Add all-to-all benchmark:
>> Example:
python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.bandwidth_test --size_mb=256Tutel v0.3.2: Add tensorcore option for extra benchmarks / Extend the example for custom experts / Allow NCCL timeout settings:
>> Example of using tensorcore:
python3 -m tutel.examples.helloworld --dtype=float32
python3 -m tutel.examples.helloworld --dtype=float32 --use_tensorcore
python3 -m tutel.examples.helloworld --dtype=float16
python3 -m tutel.examples.helloworld --dtype=float16 --use_tensorcore
>> Example of custom gates/experts:
python3 -m tutel.examples.helloworld_custom_gate_expert --batch_size=16
>> Example of NCCL timeout settings:
TUTEL_GLOBAL_TIMEOUT_SEC=60 python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.helloworld --use_tensorcore
Tutel v0.3.1: Add NCCL all_to_all_v and all_gather_v for arbitrary-length message transfers:
>> Example:
# All_to_All_v:
python3 -m torch.distributed.run --nproc_per_node=2 --master_port=7340 -m tutel.examples.nccl_all_to_all_v
# All_Gather_v:
python3 -m torch.distributed.run --nproc_per_node=2 --master_port=7340 -m tutel.examples.nccl_all_gather_v
>> How to:
net.batch_all_to_all_v([t_x_cuda, t_y_cuda, ..], common_send_counts)
net.batch_all_gather_v([t_x_cuda, t_y_cuda, ..])Tutel v0.3: Add Megablocks solution to improve decoder inference on single-GPU with num_local_expert >= 2:
…Tutel v0.2: Allow most configurations to be dynamic switchable with free cost:
…Tutel v0.1: Optimize the Einsum Complexity of Data Dispatch Encoding and Decoding, add 2DH option to deal with All-to-All at scale:
>> Example (suggest enabling 2DH only at scale, note that the value of --nproc_per_node MUST equal to total physical GPU counts per node, e.g. 8 for A100x8):
python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.helloworld --batch_size=16 --use_2dh* Prepare Recommended Pytorch >= 2.0.0:
# Windows/Linux Pytorch for NVIDIA CUDA >= 11.7:
python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
# Linux Pytorch for AMD ROCm >= 6.2.2:
python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2.2
# Windows/Linux Pytorch for CPU:
python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu* Option-1: Install Tutel Online:
$ python3 -m pip uninstall tutel -y
$ python3 -m pip install -v -U --no-build-isolation git+https://github.com/microsoft/tutel@main
* Option-2: Build Tutel from Source:
$ git clone https://github.com/microsoft/tutel --branch main
$ python3 -m pip uninstall tutel -y
$ python3 ./tutel/setup.py install --user… $ python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.helloworld --batch_size=16…Documentation for checkpoint conversion has been moved here.
…You can consult this paper below to get to know more technical details about Tutel:
@article {tutel,
author = {Changho Hwang and Wei Cui and Yifan Xiong and Ziyue Yang and Ze Liu and Han Hu and Zilong Wang and Rafael Salas and Jithin Jose and Prabhat Ram and Joe Chau and Peng Cheng and Fan Yang and Mao Yang and Yongqiang Xiong},
title = {Tutel: Adaptive Mixture-of-Experts at Scale},
year = {2022},
month = jun,
journal = {CoRR},
volume= {abs/2206.03382},
url = {https://arxiv.org/pdf/2206.03382.pdf},
}…This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.
暂无开放 Issues，或尚未同步最近议题。