Multi-LoRA training: resident adapters and mixed-adapter steps on one base model

Summary

verl trains one LoRA adapter per training engine: get_peft_model under the default adapter name, max_loras: 1 on the vLLM side, the LoRA helpers (collect_lora_params, layered_summon_lora_params) read the default adapter only, and the checkpoint path records one adapter's rank and alpha. #4421 asked for several adapters on one base during training and was closed by the scheduled cleanup task with an invitation to open a new issue with more detail. I asked there on 2026-09-10 to reopen it (https://github.com/verl-project/verl/issues/4421#issuecomment-5624318988) and haven't heard back, so here is the new issue, with what I've learned prototyping it since and a proposal to write this up as an RFC, in the shape #7579 is under review in.

The use case and the one-slot mechanism are in that comment. In one line: many adapters training against one shared frozen base on one pod, so the swap dominates and every other adapter waits. vLLM already keeps many adapters resident and batches across them for sampling (verl's docs point users at SLoRA and CCoE for serving multiple adapters, docs/advance/ppo_lora.rst); the training side has neither property.

Two levels of support: resident adapters, then batched steps

Resident: N adapters' weights, gradients and optimizer state live on the GPU at once, and steps may still run one adapter at a time. This removes the swap for the resident set. The mechanism is one FSDP2 unit per adapter per decoder layer, so an adapter that isn't routed in a step is never all-gathered (all ranks have to run the active adapters in the same order, including ranks that hold no tokens for one of them, which the design enforces); adding an adapter into an already-wrapped model worked in the prototype; and per-adapter optimizer parameter groups with a rebound step keep one adapter's step from touching another adapter's accumulated gradient.

Batched: one forward and backward over the shared base for a batch whose samples belong to different adapters. PEFT's Linear layer already supports this at the layer level (adapter_names in the forward), though its model-level API refuses it in training mode, so this needs a training path of its own. The expert-LoRA path for MoE models (ParamWrapper) refuses it, so batching expert LoRA needs a grouped GEMM with a backward; torch.nn.functional.grouped_mm is the candidate primitive; whether its autograd covers the (segments x rank) shapes at useful speed is an open question that needs further research. Punica's SGMV kernels have no backward.

The correctness target for the batched step is agreement with serialized per-adapter steps to floating-point tolerance. That means loss normalization, gradient clipping and accumulation, optimizer and scheduler state, and policy versions all stay per adapter, and anything the RL loop or the MoE path reduces over the whole batch gets split by adapter first.

What prototyping it showed

I've been prototyping both properties on public Qwen3 dense and MoE checkpoints with plain torch, transformers and PEFT on 8 GPUs. Residency works as designed and needs no new kernel: adapters register as their own FSDP2 units, globally inactive ones are never gathered, and a single-adapter step runs exactly as it does today. That alone removes the swap.

The batched step needs one more piece. Its LoRA delta is "for each segment of tokens in the batch, multiply by that segment's adapter", which is a grouped GEMM. PEFT's existing adapter_names forward is not that: it gathers and scatters per adapter at every layer and comes out slower than running the adapters one after another. torch.nn.functional.grouped_mm is the natural primitive and already has autograd; whether it's fast at LoRA-rank shapes is the open question, and if it isn't, we'd attempt to write a Triton forward and backward along the lines of vLLM's Punica kernels. For MoE expert LoRA the same grouped GEMM gets a second grouping key (expert and adapter). So the plan is residency with serialized per-adapter steps first, then the batched step once the kernel question is settled. On the training side the closest prior art I know of is mLoRA, tLoRA and ALTO, which batch concurrent LoRA jobs with their own kernels and schedulers; the RFC would position against them.

Where it would land

Opt-in verl.experimental.multi_adapter in four reviewable units: engine residency, mixed-adapter execution, rollout plumbing (N adapters resident in vLLM), and V1 trainer plumbing (a tenant key on samples). Changes outside the package are small and would be enumerated in the RFC; the single-adapter path is untouched when the feature is off. Adding an adapter into an already-wrapped model at runtime worked in the prototype, but needs a re-run of FSDP2's root lazy initialization after the first forward, which is private surface and one of the questions I'd put to maintainers; registering the resident set before the first forward is the safe alternative. Megatron is a potential follow-up. Adjacent work I'd want to line up with: #7484 (LoRA+ and MoE LoRA export on Megatron), #7749 (a BF16 LoRA training recipe on FSDP2), #7483 (LoRA for DeepSeek V4 and GLM 5.3 Flash, which fixes the vLLM LoRA weight-sync path) and #7495/#7496 (the LoRA sync bug and fix in the same path the rollout unit extends), #5790 (agent abstractions and the trajectory gateway) and the uni-agent track on the 26Q3 roadmap (#6985), where several agents over one base come up, and #7579 (the tenant key composes with its staleness filter).

Ask

If maintainers are open to this direction, I'll write the RFC with the measurements in full and post it here. If there's a shape you'd rather it took (in-tree engine change instead of the experimental package or a narrower first step), I'd rather hear that before writing it.