A model is a set of weights. An inference engine is the software that turns those weights into a service: it decides which requests are processed together, how the model's working memory is allocated, and what happens when demand arrives faster than the hardware can answer. Most of what a buyer is really choosing in a private AI deployment is that service layer.
This guide explains one mechanism inside it. Prefix reuse is the practice of keeping the work an engine has already done for a repeated block of input, so it does not have to be repeated again.
What an inference engine does
A language model produces output one token at a time, in two phases. First, prefill: the engine reads the prompt and builds the internal state the model needs in order to continue. Then decode: the model generates output tokens one after another, carrying that state forward. The state is the key-value cache, usually shortened to KV cache. For every token already processed, the model holds the intermediate values it will need again.
An inference engine manages that cache. It decides how much memory the cache may occupy, how requests are batched together, and what is discarded when memory runs short. SGLang is an open-source serving framework for large language and multimodal models, and its runtime organises the cache as a radix tree so that requests sharing a prefix can reuse the same entries. The project calls this RadixAttention and describes prefix caching as a core runtime feature; its documentation sets out the eviction and session controls in detail. We use SGLang in our own inference work, and this article is about the mechanism and the conditions around it, not about measured results of ours.
A conceptual repeated-prefix example
Imagine a support team serving one model behind an internal assistant. Every request carries the same long opening block: who the assistant is, what it must not do, the product catalogue, the tone guide. That block is identical in every request. Only the last few lines, the actual question, differ.
Without reuse, the engine prefills the whole prompt each time, including the block it has already processed hundreds of times. With prefix reuse, the engine keeps the key-value state for the shared block in its cache. When the next request arrives with the same beginning, the engine matches the cached prefix and computes only the part that is new.
That is the whole idea. It is not a different model, and it does not change the answer. It changes how much work the same answer costs. A conceptual diagram of two requests arriving with an identical opening block and diverging only at the end illustrates the shape of it; that illustration is not a measurement.
What reuse may change
The clearest effect is on prefill work. When a large part of a prompt is already cached, the engine has less new input to process before it can start generating. That can reduce the wait before the first token appears, and it can free capacity that would otherwise go into repeated prefill. Neither effect is guaranteed, and both depend on the request pattern.
Version 0.5.20 of SGLang, published on 18 September 2026, includes a cache change that the release notes describe as branching-point caching for the sliding-window-attention component of its unified radix tree: the state is kept at the point where requests fork from a shared prefix, so branches reuse it instead of recomputing. The notes give one worked example, on DeepSeek-V4-Flash with a shared system prompt, reporting the token hit rate rising from 43.8 per cent to 60.8 per cent and mean time to first token falling from 1.57 s to 1.07 s.
Those are the project's measurements, on one model, one request pattern and its own test conditions. They are a reason to test the idea against your traffic, not a result that transfers to it. The same release adds an opt-in external linker that lets the radix tree address a shared global memory pool, and a CPU-only simulator that the project says predicts time to first token within about 6 per cent on most of the serving traces it was checked against, and prefix reuse within 0.05 percentage points. A simulator is a useful way to study cache behaviour without GPUs; it is not evidence about your throughput.
Warm and cold runs, concurrency and latency
Prefix reuse has a start-up shape. The first request carrying a given prefix has to compute it; only later requests can reuse it. A test that reports a large gain after a warm-up is describing a warm state, and a service that restarts, redeploys or evicts its cache lives closer to the cold state. Both numbers are real. They answer different questions, and a single figure rarely tells you which one you are looking at.
Concurrency matters as much. Reuse reduces repeated prefill work, which helps most when many requests share prefixes and arrive together. But the cache is not free memory. SGLang's documentation describes what happens when the key-value pool is full: the cache evicts prefixes according to a policy, with least-recently-used as the default and alternatives that favour frequently matched, historically valuable or higher-priority prefixes. The same documentation describes a policy aimed at agentic and multi-turn work, where the tail of a conversation matters more than the raw hit rate, and notes that it requires the unified radix cache.
Reading order matters too. A service can improve its average time to first token while the slowest requests in a busy period get slower, because the cache competes for the same memory as everything else. Ask for the response-time distribution, not only its mean.
Operational checks before adoption
Four questions separate a useful hypothesis from a configuration choice.
- Do your requests actually share prefixes? Long fixed instructions, a shared reference document, a common system prompt or a stable tool schema give the cache something to reuse. A stream of unrelated questions does not. The mechanism cannot create a shared prefix that your workload never sends.
- Which state is reused, and under which eviction rule? SGLang's session-aware radix cache registers reusable entries to a session and evicts unreferenced entries before referenced ones. Its own documentation is explicit about the limit: session references are soft protection, not memory pins, so referenced entries can still be evicted when reclaiming everything else is not enough. Design for the possibility that a prefix you expected to be warm is gone.
- How will you measure the difference? Fix the model, precision, engine version, hardware and concurrency. Run cold and warm. Measure time to first token, generation time and throughput. Repeat the pattern you actually expect rather than a convenient one.
- Who owns the operating consequence? Cache settings, memory fractions and engine versions belong to the team that runs the service. A release that changes cache behaviour or a documented interface is a change to that service, and it needs an acceptance check and a rollback route before it is applied.
What a request pattern is worth
Prefix reuse is a good example of an optimisation that is real, documented and workload-specific. It rewards a buyer who can describe the request pattern and a team that can measure it. It disappoints anyone who adopts it because a release note said it was faster.
Before you size a private inference deployment, write down the pattern: how many requests share an opening block, how long the prompts are, how many arrive at once, and how long a user may wait. Those four answers decide whether cache behaviour is a design lever or a footnote.
LLM Machines configures and maintains private AI inference environments inside infrastructure your business controls, including model serving, access policy, monitoring, controlled updates and recovery. If your team is evaluating a private deployment, bring the request pattern and its acceptance criteria to the first technical conversation.