#8458·DeepSpeed

[BUG] Do not silently ignore use_shared_prefill with continuous batching

Author: nathon-leeCreated Sep 8, 2026Updated Sep 17, 2026
Labelsbugtraining

Context

HybridEngineRollout supports both shared prompt prefill and continuous batching, but their interaction is currently undefined.

When continuous batching is enabled through SamplingConfig.continuous_batch_size, the continuous path is selected before the shared-prefill logic. As a result, HybridEngineRolloutConfig.use_shared_prefill is silently ignored.

Continuous batching currently requires n_samples_per_prompt=1, so shared prefill does not provide a clear benefit in this mode. However, silently ignoring the configuration can be confusing and may hide an invalid combination of options.

Proposed work

Define and document the behavior of use_shared_prefill when continuous batching is enabled.

Possible choices include:

  • explicitly reject the combination with a clear ValueError; or
  • define and implement compatible shared-prefill semantics for continuous batching.

Acceptance criteria

  • The interaction between use_shared_prefill and continuous batching is explicitly defined.
  • The configuration is not silently ignored.
  • Tests cover both compatible and incompatible configurations.
  • The default non-continuous generation path remains unchanged.

Follow-up from DeepSpeed PR #8368. cc @delock