#57383 [RFC]: Asymmetric P/D Deployment for DeepSeek-V4.1 Flash

Summary.

DeepSeek-V4.1's bounded-replay structure creates an opportunity to make Prefill/Decode disaggregation substantially more asymmetric.

Today, a Prefill worker still loads the full 40-layer model even though, after layer 20, only the final 128 prompt tokens need to be processed. This keeps almost half of the model weights resident on P for a small late-layer workload whose MoE utilization is likely poor.

This RFC proposes moving that final 128-token window entirely to the Decode worker:

For a prompt of length N, P computes only [0, N-128) and transfers the resulting prefix KV state.
D computes [N-128, N) as its first full-model step and samples the first output token.
The request then continues through normal speculative decode with q_len = 1 + num_spec_tokens.

The most important benefit is deployment density: P would load only the causal encoder plus the minimal layer-20 KV/Indexer projection path, potentially removing close to half of its transformer-layer weights. A key target is fitting a production Prefill replica on one B300 GPU.

The execution model on D also becomes simpler than layer-local decoder replay. D never needs separate encoder and decoder token batches:

first D step:   all L0-L39 process the same 128 prompt tokens
later D steps:  all L0-L39 process the same 1 + num_spec_tokens tokens

The trade-off is that each newly admitted request adds a bounded-prefill step on D, which can temporarily increase Decode token count and affect ITL. The proposal is that this cost is acceptable for DeepSeek-V4.1 Flash because its compressed KV cache supports large Decode batches, while the P-side memory saving is substantial.

This RFC is a design proposal for discussion. I do not currently plan to implement it, and there is no prototype or benchmark result yet. Implementation ownership is open.

Proposed Change.

Model placement and token ownership

flowchart LR
    I[Prompt: N tokens]

    I -->|Prefix: N-W tokens| P
    I -->|Final window: W tokens| D

    subgraph P[Prefill worker: partial model]
        PE[Causal encoder L0-L19]
        PK[Minimal L20 Main-KV / Indexer<br/>projection path]
        PE --> PK
    end

    subgraph D[Decode worker: full model]
        BR[First step: bounded prefill<br/>L0-L39, q_len = W]
        FT[Sample first output token]
        ND[Later steps: speculative decode<br/>L0-L39, q_len = 1 + num_spec_tokens]
        BR --> FT --> ND
    end

    P -->|Prefix Main / Indexer KV| D

For prompts no longer than W, P performs no model forward and D processes the complete prompt.

Conceptually, P would retain:

embeddings and complete layers L0-L19;
the minimal layer-20 projections needed to produce its canonical C1 Main KV and Indexer K; and
the state required by the KV connector handoff.

P would not retain the complete layer-20 attention/MoE path, layers 21–39, final normalization, or the LM head.

Request flow

sequenceDiagram
    participant P as Prefill worker
    participant K as KV connector
    participant D as Decode worker

    P->>P: Run prompt prefix [0, N-W)
    P->>P: Produce prefix Main/Indexer KV
    P->>K: Send prefix KV and continuation state
    K->>D: Install remote prefix state
    D->>D: First step: run final W prompt tokens through L0-L39
    D->>D: Sample first output token
    D->>D: Later steps: normal speculative decode

Moving first-token generation to Decode is intentional.

Uniform Decode execution

Decode has two full-model step shapes:

new request, first step:
  q_len = W_i, where W_i <= 128
  active layers = L0-L39

later speculative decode steps:
  q_len = 1 + num_spec_tokens
  active layers = L0-L39

The first step is a bounded prefill using the transferred prefix state. After sampling, the request joins normal speculative decoding.

This avoids changing the token set after layer 20. The attention history differs by cache group, but the query rows do not:

prefix-cacheable Main/Indexer groups read the transferred prefix state;
SWA groups are rebuilt locally over the replay window, following the mechanism in #56227; and
every layer executes the same W query rows in the first D step.

If bounded-prefill requests and ongoing decode requests are scheduled together, their request-level q_len values differ. vLLM may schedule them separately or use its mixed/piecewise execution mechanisms. This is a scheduler choice rather than a model-internal layer split.

Why This May Help

Smaller Prefill replicas

P no longer needs the complete decoder stack or LM head. This is expected to remove close to half of the transformer-layer weights from each Prefill replica.

The exact saving depends on embeddings, KV pools, connector buffers, CUDA graphs, and allocator headroom. An important deployment target is fitting one production Prefill replica on a single B300 GPU.

Avoid a low-density late-decoder tail on P

With decoder-side compaction, layers 21–39 process at most 128 tokens per Prefill request. Prefill batches are normally small, so the late MoE layers may see low token and expert occupancy.

In this proposal, D executes the final window as an ordinary full-model bounded-prefill step. It can batch multiple newly admitted requests and potentially coordinate them with the larger Decode workload.

Simpler model execution

The split is between workers and token ranges, not between layer ranges inside one forward:

P handles the long prefix.
D handles the bounded final window.
Every D forward uses one token set across all layers.

Chunked Prefill and context parallelism apply to P's prefix. D receives ordinary token IDs for the final window, without an internal-layer activation handoff.

Cost and Trade-offs

The main cost is a larger first step on D:

new request, first step:  q_len <= 128
later steps:              q_len = 1 + num_spec_tokens

Potential costs include:

D must support both bounded-prefill and speculative-decode shapes.
Admission bursts may increase ITL/TPOT for running requests.
Mixing both request types in one scheduler iteration may require piecewise CUDA graphs or eager fallback.
P/D must preserve compressor continuation state and a consistent split boundary.

This may be acceptable for DeepSeek-V4.1 Flash because its compressed KV cache permits a large Decode batch, and steps admitting new requests are usually a small fraction of all Decode steps. This requires measurement.

Feedback Period.

Relationship to Existing vLLM Work

This proposal builds on, but is different from, two open vLLM PRs.

#56227: encoder-side SWA bounded replay

#56227 already provides much of the required runtime behavior:

after a local or KV-connector prefix hit ending at H, the scheduler rewinds the final window;
the worker recomputes [H-W, H) locally;
Main/Indexer KV remains cached or transferred, while SWA is rebuilt locally; and
Request.replay_start and attention metadata floor the replay window.

This demonstrates that a worker can adopt remote Main/Indexer KV and rebuild the final SWA window locally.

The difference is that #56227 currently treats replay as a consequence of a prefix hit. This RFC proposes making that replay boundary the P/D split itself: P intentionally stops at N-W, and D intentionally owns the final W prompt tokens.

#56752: decoder-side SWA bounded replay

#56752 compacts the batch after layer 20 so layers 21–39 process only each request's trailing window.

That design still performs one forward in which different layer ranges process different token sets:

L0-L20:  full prompt rows
L21-L39: trailing W rows

This RFC proposes a different deployment boundary:

P: [0, N-W)
D: [N-W, N), through all L0-L39

Within a Decode step, every layer processes the same rows. There is no layer-20 compaction boundary, internal gather/scatter, or replay-layer sub-batch.

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.