Transformers v5 MoE conversion incorrectly rewrites fully qualified shared-expert LoRA targets

System Info

Environment

• Model: amd/Instella-MoE-16B-A3B-Think
• PEFT: 0.20.0
• Transformers: 5.16.1
• trust_remote_code=True

Who can help?

No response

Reproduction

Problem

Instella contains two different representations of MoE projections:

1. Routed experts use fused 3D parameters, for example:

   model.layers.12.mlp.experts.down_proj

2. Shared experts are regular MLP modules containing nn.Linear layers, for example:

   model.layers.12.mlp.shared_experts.down_proj

A mixed LoRA configuration therefore legitimately uses both target_parameters and target_modules:

config = LoraConfig(
    r=8,
    target_modules=[
        "model.layers.12.mlp.shared_experts.down_proj",
    ],
    target_parameters=[
        "model.layers.12.mlp.experts.down_proj",
    ],
)

peft_model = get_peft_model(model, config)

Expected behavior:

• model.layers.12.mlp.shared_experts.down_proj is retained as a module target.
• model.layers.12.mlp.experts.down_proj is retained as a parameter target.
• PEFT installs a regular LoRA layer on the shared expert and a parameter wrapper on the routed expert bank.

Actual behavior:

config.target_modules == set()
config.target_parameters == {"down_proj"}

Only routed-expert parameter wrappers are created. The shared-expert nn.Linear module is not adapted. The fully qualified, layer-specific target is also reduced to the generic name down_proj, potentially broadening it to other matching expert parameters.

Cause

Instella reports model_type="deepseek_v3". When Transformers v5 conversion is enabled, PEFT applies the deepseek_v3 -> qwen2_moe checkpoint conversion mapping. The MoE conversion logic treats any target module equal to, or ending in down_proj as an old routed-expert target. It does not check whether the path refers to .experts. or .shared_experts..

Consequently, this ordinary linear module:

model.layers.12.mlp.shared_experts.down_proj

is mistaken for a fused routed-expert parameter. It is removed from target_modules, and only its leaf name, down_proj, is added to target_parameters.

This conversion is also applied to newly created LoraConfig objects passed to get_peft_model; it is not limited to loading legacy adapters.

Relevant model files:

• https://huggingface.co/amd/Instella-MoE-16B-A3B-Think/blob/main/config.json
• https://huggingface.co/amd/Instella-MoE-16B-A3B-Think/blob/main/modeling_instella_moe.py

Relevant PEFT conversion logic:

• https://github.com/huggingface/peft/blob/main/src/peft/utils/transformers_weight_conversion.py

Reproduction Code:

from types import SimpleNamespace

import torch
from peft import LoraConfig
from peft.utils.transformers_weight_conversion import (
    convert_peft_config_for_transformers,
)

# Instella declares model_type="deepseek_v3".
model = torch.nn.Module()
model.config = SimpleNamespace(model_type="deepseek_v3")

config = LoraConfig(
    r=8,
    target_modules=[
        "model.layers.12.mlp.shared_experts.down_proj",
    ],
    target_parameters=[
        "model.layers.12.mlp.experts.down_proj",
    ],
)

print("Before:")
print("  target_modules:   ", config.target_modules)
print("  target_parameters:", config.target_parameters)

convert_peft_config_for_transformers(config, model, conversions=[])

print("After:")
print("  target_modules:   ", config.target_modules)
print("  target_parameters:", config.target_parameters)

Expected behavior

The conversion should distinguish routed expert paths from ordinary/shared MLP paths. Possible approaches:

• Apply the MoE module-to-parameter conversion only to targets structurally inside a routed .experts. container.
• Preserve fully qualified paths rather than reducing them to the final component.
• Convert only known legacy routed-expert path forms, while leaving .shared_experts.* targets untouched.
• Inspect whether the resolved target is an nn.Module or a direct fused parameter before changing target_modules into target_parameters.

Bare targets such as "down_proj" may need the existing broad behavior for backward compatibility, but a fully qualified .shared_experts.down_proj target should not be interpreted as a routed-expert bank.