Qwen3 MoE Error

Author: ehristoforuCreated Sep 7, 2025Updated Feb 4, 2026

My config:

base_model: Qwen/Qwen3-1.7B
architecture: qwen
gate_mode: hidden
experts_per_token: 2
dtype: bfloat16
experts:
  - source_model: Qwen/Qwen3-1.7B
    positive_prompts:
    - "chat"
    - "assistant"
    - "tell me"
    - "explain"
  - source_model: Qwen/Qwen3-1.7B-Base
    positive_prompts:
    - "write a story"
    - "compose an email"
    - "create a blog post"
    - "generate a report"

Output:

[2025-09-07 16:16:40] [INFO] Running mergekit-moe config.yaml merge --copy-tokenizer --allow-crimes --lazy-unpickle --lora-merge-cache /tmp/tmpihbkppto/.lora_cache --out-shard-size 5B
[2025-09-07 16:16:44] [INFO] ERROR:root:No output architecture found that is compatible with the given models.
[2025-09-07 16:16:44] [INFO] ERROR:root:All supported output architectures:
[2025-09-07 16:16:44] [INFO] ERROR:root:  * Mixtral
[2025-09-07 16:16:44] [INFO] ERROR:root:  * DeepSeek MoE
[2025-09-07 16:16:44] [INFO] ERROR:root:  * Qwen MoE

For some reason, the merge of Qwen3 models in Qwen3MoE is not detected. I also tried with shared_experts - the same thing. I use the latest version of mergekit (branch main).

Source: arcee-ai/mergekit

View original on GitHub View discussion on GitHub