Picture a private assistant used by two teams. One team asks short questions all day: where is order 4471, summarise this ticket. Another team pastes a 120,000-token contract and asks for a risk summary. On a quiet day both feel fast. At ten in the morning, six contracts arrive while the short questions keep coming, and the short questions start to wait.
The model has not become worse. The queue has changed. Long prompts are expensive in a different way from long answers, and an inference engine has to decide which work gets the machine next. Inference here means running a model to produce an output.
Two phases, two kinds of work
The work happens in two phases. Prefill reads the prompt and builds the key-value cache, the stored state the model needs to continue. That phase is compute-heavy and grows with input length. Decode then produces one token at a time, reading that cache on every step. Decode is memory-bandwidth-heavy and grows with the number of tokens generated and with how many requests are in flight.
A short question and a 120,000-token contract use the same engine but not the same way. The contract spends a long time in prefill before its first token appears. The short question clears prefill quickly and then wants steady decode.
What goes wrong in a shared queue
In a single engine, prefill and decode batches are scheduled together. A large prefill arriving during decode can interrupt the decode batch, so tokens already being generated pause. Under data-parallel attention there is a second effect: one worker can be processing a prefill while another is decoding, and the decode side slows down.
The result is the morning above. The short questions are not short in the queue, because they are waiting behind work that has not produced a token yet.
Splitting the two stages
Prefill and decode can run in separate pools, an arrangement SGLang calls prefill-decode disaggregation. Prefill workers do the prompt work, decode workers generate, and the key-value cache moves between them over the network. Each pool can then be sized and tuned for its own bottleneck instead of compromising.
The cost is visible in the architecture: a second pool to operate, a cache transfer on the network, and a topology that has to be planned rather than assembled. The documentation lists Mooncake and NIXL as transfer engines, and the transfer path has to keep up with the prefill rate.
The change in SGLang 0.5.22
SGLang's release on 19 October includes an opt-in change to how decode reserves memory in that split arrangement. By default, decode reserves destination key-value slots for a request while its prefill is still running. A long prefill can therefore hold decode capacity that cannot yet be used. The new policy, enabled with --disaggregation-decode-allocation-policy prefill_complete, waits until the final prefill result is accepted before decode reserves the slots.
The release describes the effect as up to 27 percent more throughput and half the p95 time to first token on long-prompt workloads. That is the maintainers' measurement, not ours, and the pull request behind it is more informative than the headline.
What the upstream test shows
The test used GLM-5.2-NVFP4 on 12 Rubin GPUs, with two prefill replicas and one decode replica, each TP4/DP4/EP4, FP8 key-value cache and Mooncake. Decode key-value capacity was deliberately capped at 163,840 tokens per rank to create pressure. The mixed workload started with 64 short requests at 4,096 input tokens and 256 output tokens, concurrency 8, then ran 256 requests: 64 at 131,072 input tokens and 192 at 4,096, all at 256 output tokens and concurrency 64.
Across two runs per policy, the mean of output tokens per second rose by about 27 percent and p95 time to first token fell by about 49 percent. Mean time per output token rose by about 2 percent. The number that matters most for a streaming interface is the one that got worse: p99 inter-token latency rose by about 93 percent.
The authors are explicit that the comparison is arithmetic means of two runs, not pooled percentiles or confidence intervals, and that a separate Qwen3-8B test showed differences of about 1 percent within run variation. They also say deferred allocation does not speed up allocation itself or the decode kernels. It changes who waits.
Why the tail matters
A p95 time to first token is what a user feels before any text appears. Inter-token latency is what they feel while reading. A streaming answer that pauses every so often is worse than a slightly slower but even one for a person reading along, and worse still for an application that passes tokens to a downstream step.
That trade is not a defect. It follows from the change: holding decode capacity back until prefill completes keeps some requests waiting for slots while others start sooner. If your workload is mostly short prompts, or your users read whole answers after they arrive, the p99 tail may not matter much. If they watch the text appear, it does.
What to check before enabling it
Start with the request mix. How many prompts are long enough to spend meaningful time in prefill? A service where 2 percent of requests carry 60 percent of the tokens is a different case from one where almost every request is short.
Then measure the two waits separately, by prompt-length bucket: time to first token and inter-token latency. A single average across all requests will hide both the long-prompt cost and the short-request tail. Record concurrency, output length, model, precision, engine version and transfer backend beside the numbers, because none of the figures move without them.
Check the capacity of the decode pool, not only the prefill pool. The policy targets exactly the case where decode slots are scarce. If decode has headroom, the change has less to fix.
Finally, check the constraints in the documentation. The initial implementation requires Mooncake, including its TCP mode, the Python bootstrap registry and attention tensor or context parallelism of one per rank, with attention data parallelism allowed when each rank has attention tensor parallelism of one. Decode radix cache, decode HiCache, staging buffers and several other configurations are excluded. Both roles must run the same policy, and all workers should be upgraded with the old policy in place before switching.
The decision
PD disaggregation changes which request sets the pace. It is a way to stop one kind of request from defining the experience of another, and the 0.5.22 change makes that separation more useful when long and short prompts share a queue. The upstream result is a real gain on two metrics and a real cost on a third.
The useful question for a buyer is not whether the feature is in the release notes. It is which request you are willing to make wait, and whether the metric that gets worse is the one your users notice. The figures above are the maintainers'. The next step is a test against a named workload, with the two waits measured separately.