A benchmark score is a reason to investigate a model. It is not evidence that the model belongs in your environment. The score was produced by someone else's tasks, under someone else's conditions, and it usually arrives without the workload detail a purchasing decision needs.
This guide sets out five checks that turn a published result into a defensible shortlist. The worked example is the 7 September 2026 Artificial Analysis Intelligence Index v4.3 update. It is useful precisely because it changes an evaluation rather than a product: it shows how a score moves when the test behind it changes, and why the evaluation version belongs beside any number used in a buying decision.
What an evaluation change means
On 7 September 2026, Artificial Analysis announced Intelligence Index v4.3. In its announcement, the update upgrades Terminal-Bench to v4.0 and replaces tau-3-Banking with AutomationBench-AA, an agentic workflow benchmark run on a held-out test set in collaboration with Zapier. The share of index weighting that comes from private, held-out test sets rises from 40 per cent to 45 per cent. The category weights are unchanged: agents 30 per cent, coding 20 per cent, general 30 per cent and scientific reasoning 20 per cent.
That follows the 4 September v4.2 update, which added AA-Briefcase and GDP.pdf, removed GPQA Diamond and raised the private-test weighting to 40 per cent. Two versions changed the composition of the same index in four days.
Artificial Analysis also reports that equal headline scores can carry very different cost and token profiles under its own measurement assumptions. In its v4.3 results, Claude Fable 5.1 (max with fallback) and GPT-6 Astra (max) both score 53, with an average cost per index task of USD 7.63 and USD 3.26 respectively. GLM-5.3-Flash and GPT-5.6 Terra (max) both score 42, at USD 0.25 and USD 1.40 per task. Its separate 9 September report on GPT-6 Astra puts that model at about 27,000 output tokens per task at maximum effort, against 78,000 for Claude Fable 5.1 at the same score.
Those are Artificial Analysis' measurements under its published conditions. They are not our results, not a price your business would pay, and not a recommendation. They illustrate one point: a score is comparable only inside the version and the conditions that produced it.
1. Define the representative workload first
Write down the work the model is expected to do before comparing any model. Which tasks, how often, how many requests at once, how large the inputs are, how long a user may wait, and during which hours the service must hold. A summarisation batch, an interactive assistant and an agent that calls external tools have different requirements even when they run on the same hardware.
This is also the only way to answer the question a score cannot: what would good enough look like for us? Agree a measurable threshold, such as an acceptance rate, a review time or a containment rate, before the comparison starts, so that the evaluation has something to pass or fail.
2. Check model and serving constraints
A model is deployable in your environment only if the serving path supports it. Check the runtime's own documentation for the model family, the precision you intend to run and the hardware path, and read any stated limitation there rather than in a summary. Upstream support is a starting condition, not a promise for your configuration, and it says nothing about your quality threshold.
Weight format, context length and memory behaviour decide whether a candidate fits the infrastructure you are prepared to operate. Those checks are cheaper before a shortlist is agreed than after a proposal has been written around a model that does not fit.
3. Compare quality with latency and operating requirements
A quality result without a response-time condition is incomplete. Time to first token, generation speed and the number of concurrent users describe the same system, and an improvement in one can cost another. When a benchmark or a vendor publishes a throughput figure, ask which response-time condition it was measured under.
Add the operating questions at the same time: who approves a model or runtime update, what the accepted configuration is, and how an earlier supported configuration would be restored if a change fails its checks. These are configuration and maintenance decisions, and they belong in the shortlist rather than in the week after go-live.
4. Keep the benchmark's scope visible
Record the evaluation name, version and date next to every score you reuse, together with the fact that it measures the publisher's tasks. An index that changes its agentic evaluations, its weighting or its test-set mix is not a stable ruler; a score from one version should not be compared with a score from another without saying so.
The same discipline applies to your own tests. Keep the model, runtime and configuration with the result. A number without its conditions cannot be reproduced, and an unreproducible number cannot be reviewed by anyone else in the business.
5. Build the shortlist with explicit unknowns
End with a shortlist that admits what is still unproven. Give every unknown an owner and the test that would resolve it. An unknown with an owner and a date is a task. An unknown without one is a risk the business has chosen to accept, and it should be recorded as such.
The five-column worksheet
Use five columns for the review:
- Quality - the threshold the business will accept and how it will be demonstrated.
- Workload - representative tasks, volume, input size and concurrency.
- Deployment - where inference runs, who operates it and what access is approved.
- Resources - the serving configuration, precision, memory and expected growth.
- Operations - update approval, monitoring, recovery and who owns each of them.
Each row should carry an owner and the evidence the business will accept. Leave unresolved entries visible until someone owns the next action.
What we do with this ourselves
In our own inference work we use SGLang, and its supported-model list is where a compatibility check starts rather than an assumption that a model is servable. LLM Machines configures and maintains private inference on infrastructure the customer controls. The scope covers workload discovery, capacity planning, model serving and operational support, while customer-selected applications remain under customer or third-party ownership and connect through a documented inference API.
If your team is preparing a private inference decision, the most useful first step is not a model list. It is the workload and the acceptance criteria you want the evaluation to test.