GMPO: use_liger_kernel=True silently bypasses the GMPO loss; MoE aux loss and entropy bonus dropped (gaps not covered by #6095)

Description

Follow-up to #6056 and complementary to open PR #6095 (which addresses the vLLM IS correction, off-policy masking, the zero-advantage clip branch, and the loss_type default). Two remaining gaps in trl/experimental/gmpo/gmpo_trainer.py are not covered by that PR (at f72460224c5477e65031b2363f54368699105182):

1. use_liger_kernel=True silently bypasses the GMPO loss entirely

GMPOTrainer only overrides _compute_loss, but the inherited compute_loss routes around it when liger is enabled:

# trl/trainer/grpo_trainer.py (inherited by GMPOTrainer)
def compute_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):
    ...
    if self.use_liger_kernel:
        # Compute the loss using the liger grpo loss
        unwrapped_model = self.accelerator.unwrap_model(model)
        return self._forward_redirection(model, unwrapped_model, self.compute_liger_loss, unwrapped_model, inputs)
    return self._compute_loss(model, inputs)

The init-time guard in main GRPO only rejects liger for importance_sampling_level other than token/sequence, which GMPO doesn't change — so GMPOTrainer with use_liger_kernel=True silently trains with the plain GRPO clipped-surrogate loss instead of GMPO's geometric-mean objective. (gspo_token is incidentally protected by that guard via importance_sampling_level="sequence_token"; GMPO is not.) The trainer should either raise on use_liger_kernel=True or route the liger path to a GMPO implementation.

2. MoE aux loss and entropy bonus silently dropped

GMPO's _compute_loss calls _get_per_token_logps_and_entropies without compute_aux_loss=self.aux_loss_enabled and never adds the router_aux_loss_coef * aux_loss term, so MoE models lose the load-balancing loss. The _entropy_bonus_enabled block from main GRPO is also absent, so entropy_coef/adaptive-entropy settings are silently ignored. (Minor, same family: the forward kwargs are missing spatial_shapes/num_tiles for LFM2-VL-style models.)

Per the repo's consistency policy for duplicated trainer blocks, these should be re-aligned with trl/trainer/grpo_trainer.py::_compute_loss, keeping the geometric-mean objective as the intentional divergence.

I'm happy to submit a PR (rebasing on #6095 if it lands first).

Reproduction

Code inspection; for item 1, instantiate GMPOTrainer with use_liger_kernel=True and observe that GMPOTrainer._compute_loss is never called (the liger GRPO loss is used instead).

System Info

Found by code inspection (AI-assisted audit) at f72460224c5477e65031b2363f54368699105182.

• Platform: macOS-26.5.1-arm64-arm-64bit
• Python version: 3.12.7
• TRL version: 1.5.1+f724602
• PyTorch version: 2.7.1
• Transformers version: 5.8.1
• Accelerate version: 1.8.1

Checklist

• I have checked that my issue isn't already filed (see open issues)
• I have included my system information
• Any code provided is minimal, complete, and reproducible (more on MREs)
• Any code provided is properly formatted in code blocks, (no screenshot, more on code blocks)
• Any traceback provided is complete