ds4-server: expose tensor-parallel roles (field report: 2x M4 Max TB5, TP 22.7 t/s vs server-pipeline ~20)
Feature request: expose tensor-parallel roles in ds4-server
The README notes tensor-parallel roles are currently CLI-only ("not by ds4-server or ds4-agent"), and ds4-server indeed rejects --tensor-parallel (unknown option), though the shared --help distributed text lists it. This is a request to bring TP roles to the server, with a field report for motivation.
Field report (2× MacBook Pro M4 Max 128 GB, TB5, TCP transport)
DeepSeek V4 Flash 0731, Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix-0731:
| mode | decode | prefill (short) |
|---|---|---|
ds4 CLI --tensor-parallel |
22.7 t/s sustained (550-tok gen) | 45.2 t/s |
ds4-server pipelined distributed (--layers 0:20 / 21:output) |
~19-20 t/s through the OpenAI API | — |
Both work great (and the pipelined server + disk-KV is what we deployed), but TP's lower per-token latency is the better interactive experience, and today it's unreachable behind the HTTP APIs. Notes from the bring-up in case they're useful:
- M4 Max ("Metal 4 tensor API disabled for pre-M5") — numbers above are without it.
- RDMA:
rdma_ctl enablerequires Recovery OS, so everything above is--transport tcpover the TB5 bridge IPs; TP sync gates were not the bottleneck at these sizes. - Mismatched
-cbetween TP ranks fails at prefill warmup (mirrored contexts work) — a friendlier startup check for mirrored-session geometry would help. - The residency pre-fault + warm design is excellent: rank warm from page cache is ~1.6 s for an ~81 GiB shard.
If TP-in-server is on the roadmap already, happy to test early builds on this pair.
Source: antirez/ds4