Two suppliers can quote the same server, the same model and the same answer length, and still send throughput numbers that are three times apart. The gap is rarely a disagreement about the hardware. It is usually a disagreement about what the benchmark measured: how many requests ran at once, how long the prompts were, and how much of the answer was allowed to arrive after a long wait.
That is the problem an acceptance test solves. Before you compare proposals, you decide what the service has to do and how you will recognise when it does it. The test then measures your workload, not the one a vendor happened to pick.
Start from the workload, not the metric
A benchmark needs a subject. The subject is the workload: which model at which precision, how long a context one request may carry, how many requests arrive together, and how long a user may wait. Those four facts decide what any later number can mean. A model that fits comfortably for one person may not fit when the service holds a long conversation for twenty people at once, because the engine keeps the working state of every request in flight.
Write the workload down before you write the test. It is the part of the document that is easiest to leave vague, and the part a supplier will probe first.
A useful way to fix the workload is to take it from something you already run. If the service will answer customer questions, keep a sample of the real questions and note how long they are. If it will summarise documents, measure the documents rather than imagine them. A workload assembled from guesses produces a test that passes and a service that still surprises you later.
Separate the wait from the rest
One request has two phases, and they behave differently. The engine first reads the whole prompt; this is the wait until the first token appears. It then writes the answer one token at a time; that is the speed a user feels for the rest of the reply. A change that improves one often does little for the other, so a single tokens-per-second figure hides the trade-off a user actually notices.
A dated example shows why the two should be measured apart. On 29 September 2026, Artificial Analysis published an open tool that replays eight recorded agent sessions, 168 model turns, with each request carrying the whole conversation so far and the context growing to roughly 56,000 tokens. In its results, between 73% and 93% of prompt tokens were served from the KV cache, yet reading each turn's new input still took between 22% and 41% of the end-to-end time for one of the models tested. Caching helps; it does not remove the cost of the first read. These are Artificial Analysis's findings on laptop and workstation hardware at 4-bit quantisation, not ours, and not a server measurement.
Set concurrency and a response-time target together
Throughput and latency pull against each other. At low concurrency a system answers a few requests quickly; at high concurrency it serves more requests per second while each one waits longer. A throughput figure with no latency condition beside it therefore describes a different service from the one you may have agreed.
The usual way to see this is a curve rather than a single point: requests per second against the wait for a first token, one point per concurrency level. NVIDIA's GenAI-Perf guide, dated 18 June 2025, builds exactly that curve and uses it to size a deployment and estimate cost. Its examples are synthetic and its prices are illustrative; the method is the transferable part. Ask a supplier for the curve, or run it yourself, and mark the point that matches your own arrival pattern.
Your own traffic tells you where to set the concurrency. If request logs show that twenty people use the tool at once in the busiest hour, test at twenty and above, not at one and not at a hundred. The target is the busiest realistic moment with a little headroom, not the largest number the hardware can survive.
Record the conditions beside every number
A result is reproducible only if the setup travels with it. Note the model and its version, the serving engine and its version, the numeric precision or quantisation, the hardware, the context length, the concurrency, and whether the cache started warm or cold. Change any one of those and the number can move.
SGLang's v0.5.21 release, published on 2 October 2026, shows why the version matters. Its notes list specific measurements next to specific changes, including a reported 22% faster first token on long prompts for one model and a prefix cache now running on a Rust core. We use SGLang in our inference work, and these are the kind of notes we read closely. The figure is the maintainers' measurement on their configuration, and it means something only with that configuration beside it. A benchmark that does not name its versions cannot be repeated by anyone, including you.
Agree acceptance and repeatability
Decide the pass mark before the run. A quality threshold on your own prompts, a first-token target and a completion target together give the test something to fail against. If nothing can fail, the test is a demonstration.
Quality belongs in the same document. Name the rule your answers must satisfy and score a sample of real prompts against it before the run, not after. Two models that both clear the speed target may not both clear that rule, and the faster one is only the better choice if it clears both.
Then run the test twice and keep the spread. Two runs of the same workload on the same configuration show the variance your acceptance decision actually sits on. A single run is a timestamp rather than a result.
Finally, decide who signs off and what happens to the configuration if a later update moves the outcome. That is the same ownership question you will meet the day the service is in production, asked early.
Done this way, the benchmark stops being a comparison of brochures and becomes the first honest conversation about your workload. If you are sizing private inference infrastructure and want to work through the acceptance criteria with us, bring the workload and the targets.