When one GPU isn't enough: multi-GPU and P/D disaggregation

Your single-GPU server runs fine until the p99 of inter-token latency spikes. The usual culprit is a 30K-token prompt arriving while ten requests are generating: everything stalls until the prefill finishes. A single vllm serve does three jobs that get in each other's way: prefill (compute-bound, sets your TTFT), decode (memory-bound, sets your ITL) and a pile of CPU work — tokenizing, parsing tool calls, detokenizing — running on the box you rented for its GPUs. This chapter closes the guide by splitting those jobs once they can't share a GPU with dignity.

What it buys and what it doesn't

What you buy is goodput: the request rate you sustain while meeting both TTFT and ITL SLOs. With no latency target, splitting the same GPUs into prefill and decode pools won't necessarily move more tokens per second. With no 30K-token prefill pausing every decode stream, the p99 stops being a lottery.

The numbers from vLLM's official post, measured on a 2×L40S box with ~8K-token prompts: collocated, ITL p99 jumps from 23 ms to 169 ms at 0.4 req/s; with P/D separated it stays at 29 ms and never exceeds 52 ms. On an AMD 8×MI300X node, separating prefill and decode raised goodput from 30 to 73 out of 100 requests meeting both SLOs: 2.4x on the same hardware. Source: Taking vLLM Apart, September 29, 2026.

Before getting excited, the calculation that decides whether this applies to you: the KV cache travels from prefiller to decoder. Llama-3.1-70B in BF16 stores 320 KiB per token — a 10K prompt is ~3 GB crossing the network before the first token. If the transfer is slow, every first token pays the toll. First, nvidia-smi topo -p2p r: if it says NS between your GPUs, forget it on a single box. Second, check the KV Transfer metrics line in the decoder's log: if Avg xfer time sits in the hundreds of milliseconds, fix that before measuring anything else.

The minimum experiment: two GPUs and a proxy

# Prefiller on GPU 0
CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-7B-Instruct --port 8100 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'

# Decoder on GPU 1
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen2.5-7B-Instruct --port 8200 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'

# Proxy chaining them: prefill first, decode after
python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py --port 8192 \
  --prefiller-hosts localhost --prefiller-ports 8100 \
  --decoder-hosts localhost --decoder-ports 8200

The proxy's trick: send the request to the prefiller with max_tokens=1 and do_remote_decode: true. The prefiller computes the KV cache, holds it, and returns kv_transfer_params pointing at those blocks; the decoder pulls them over NIXL and generates. For production that example proxy won't do — the real orchestrators are llm-d and Dynamo.

And the honest measurement, with the same bench you've used since chapter 6 of the previous guide, now with goodput:

vllm bench serve --model Qwen/Qwen2.5-7B-Instruct --port 8192 \
  --dataset-name random --random-input-len 8192 --random-output-len 256 --ignore-eos \
  --request-rate 0.4 --num-prompts 100 \
  --percentile-metrics ttft,tpot,itl --metric-percentiles 50,99 \
  --goodput ttft:2000 tpot:30

Push --request-rate up until the collocated server's p99 collapses. That point is your architecture decision.

Multi-GPU without splitting anything

Before cutting the pipeline, the options inside a single instance: tensor parallelism (--tensor-parallel-size) shards the model's layers across GPUs; expert parallelism distributes a MoE's experts; pipeline parallelism stacks stages across nodes. For models that don't fit, TP is the default. For giant MoEs on clusters, EP. And if your MoE traffic varies, Elastic EP scales experts up and down without restarting the server — near-zero downtime, published October 2026.

Remember chapter 3: PP doesn't compose with speculative decoding. And for multi-turn agents, P/D with bidirectional_kv_xfer reuses the KV the decoder already holds from the previous turn instead of recomputing — up to 3.3x better TTFT on long turns. That said: check that your chat template doesn't drop the thinking traces between turns, because then decode's blocks no longer match the prompt and you get wrong output, not just slow output.

The decision table

Checkpoint and the typical error

Checkpoint: you have your p99 ITL number at the load that hurts, and you know whether your fabric can move KV fast. With that, the decision to split or not is arithmetic, not faith.

The typical error: benchmarking P/D with a 0.6B model. A model that small prefills so fast it barely disturbs decode — there's nothing to separate. Use a 7B or bigger, with prompts from your real workload, and measure turn 5 of a conversation, not turn 1.

Guide wrap-up

Five chapters and your server already: speaks JSON with guaranteed schema (ch. 1), fits the GPU you pay for (ch. 2), answers faster with speculation (ch. 3), survives your agents with the memory ladder (ch. 4), and splits the work when concurrency demands it (ch. 5). The next step is orchestration: llm-d or Dynamo on Kubernetes, which is a whole other journey.

What are you serving first with all this?