[功能] 独立可插拔的 MoE V2 模块 + moe_type 切换机制

▎ Branch address: https://GitHub.com/LiuCharming/minimind/tree/feature/moe-v2 ▎ ▎ Overview of changes ▎ ▎ Core changes ▎ ▎ 1. New model/moe.py — an independent MoE module (model/moe.py, about 670 lines), which can be switched between the original MoE and the new one via the moe_type configuration item, without affecting existing functionality. ▎ ▎ 2. Modified model/model_minimind.py ▎ - MiniMindConfig added a moe_type field ("v1" = original / "v2" = new MoEBlock) ▎ - MiniMindBlock automatically selects FFN/MoE-v1/MoE-v2 based on use_moe + moe_type ▎ - New MOEFeedForwardV2 wrapper class added as a drop-in replacement ▎ ▎ 3. All training scripts added CLI parameters ▎ - train_pretrain.py, train_full_sft.py, train_lora.py, train_dpo.py, train_ppo.py, train_grpo.py, train_agent.py, train_distillation.py have all been added --moe_type, --num_experts, --num_experts_per_tok ▎ ▎ 4. eval_llm.py added corresponding parameters to ensure correct loading of V2 weights during inference ▎ ▎ 5. trainer_utils.py added path resolution to fix the loading problem with the relative path of ../model in different working directories ▎ ▎ Usage ▎ ▎ # V2 MoE pre-training ▎ Python trainer/train_pretrain.py --use_moe 1 --moe_type v2 --num_experts 6 --num_experts_per_tok 1 ▎ ▎ # V2 MoE inference ▎ Python eval_llm.py --use_moe 1 --moe_type v2 --num_experts 6 --num_experts_per_tok 1 --weight pretrain ▎ ▎ Welcome to try it out and give your suggestions!