A server announcement is a headline about a chip. The purchase decision is about a workload. Between the two sit four questions the headline does not answer: will the model you chose fit in memory at the precision you intend to run, how fast can that memory be read, what has to talk to what, and can you power, cool and service the machine where it will live.
This guide is about those four checks, and about what belongs in a sizing brief.
Start with the model and the workload
Before a hardware question can be answered, three things about the workload have to exist. How large the model is, counted in the way it will actually be loaded rather than as a marketing figure. How long a context the system must hold for one request, and how many requests arrive at once. And how long a user may wait for a first token and for the answer to finish.
Everything else follows from those. A model that fits comfortably for one person may not fit when the service must hold a long conversation for twenty people at once, because the system keeps the working state of every request in flight.
Memory capacity: does the working set fit?
Capacity is the first constraint and the easiest to get wrong. What has to fit is the model's weights, the working state the engine keeps for each active request, and whatever the runtime itself needs. The figure that matters is not one number but the sum of those parts, and it moves with precision: the same model stored in a lower-precision numeric format occupies less memory, with quality consequences that have to be tested rather than assumed.
A recent, large example makes the shape of the question clear. On 23 September 2026 Supermicro said it is now shipping NVIDIA Vera Rubin NVL72 racks, and described one rack as pairing 72 Rubin GPUs with 36 Vera CPUs in a single liquid-cooled cabinet, with 20.7 TB of HBM4 memory and up to 54 TB of LPDDR5X across the rack. NVIDIA's own specification page for the same system lists the same 20.7 TB of HBM4 and the same 54 TB figure.
Those are rack-level numbers for a data-centre platform far above the size of most private inference deployments, which start with one or two nodes. They still make the shape of the question clear, because capacity is a pooled quantity: what matters is how much fast memory one machine presents to one model, not how much sits inside one chip.
Bandwidth: how fast can that memory be read?
Capacity decides whether a model fits. Bandwidth decides how quickly the system can move the data it needs while generating. Generation is memory-bound in most serving setups: producing each token requires reading weights and cache, so the rate at which memory can be read often limits throughput long before arithmetic does.
This is why two systems of the same capacity can behave differently. It is also why a vendor's comparison of its own products by memory bandwidth is worth reading as a statement about one specific configuration rather than a general property. NVIDIA's specification page for the Vera Rubin rack states 216 TB/s of NVLink bandwidth for the rack, with a per-GPU memory bandwidth figure of 19.2 TB/s; Supermicro's announcement states the same 216 TB/s for the nine NVLink switch trays connecting the rack's compute trays. Third-party summaries of the same platform circulate different per-GPU bandwidth numbers. When sources disagree on a figure, the safe move is to use the number the manufacturer publishes for the exact configuration, and to say which document it came from.
A private deployment meets the same constraint at a smaller scale. A single server asked to serve long-context requests at a demanding response time can be limited by how fast its memory subsystem feeds the model, and that limit is invisible in a list of chip names.
Interconnect: what has to talk to what?
The third question is how the parts of a system reach each other. Within one server that may be PCIe between host and accelerators. Across servers it is the network, and the difference between fast scale-up links and a general-purpose fabric becomes an architecture decision rather than a cabling detail.
Serving engines now build around that difference explicitly. The vLLM project's 0.30.0 release, published on 22 September 2026, lists among other things an opt-in FlashInfer PCIe IPC all-reduce path for machines without NVLink. That line is a reminder that the same software behaves differently depending on the interconnect it is given, and that a configuration sensible on one topology may be a workaround on another.
Precision and runtime support are part of the hardware decision
The same release notes read as a list of exact combinations rather than general capabilities. vLLM 0.30.0 describes DeepSeek-V4.1-Flash with the whole key-value store held in MXFP8 through the FlashMLA V4.1 record on SM100, an NVFP4 path that becomes the default on SM100 and SM103, a ROCm route and a LoRA route for one model, and a DeepSeek-V4 CPU backend with AVX512 and AMX kernels. None of those statements means that a model runs well on a machine. Each means that on one named hardware and numeric combination, one supported path exists.
That is the practical link between a chip headline and a deployment. Hardware support inside an inference engine is specific: a model, a precision, an accelerator generation and a software version, together. A buyer who records those four facts has something a later incident can be reasoned about from. A buyer who records a product family name does not.
Power, cooling and serviceability
Finally, the machine has to live somewhere. Available power per rack, cooling capacity, floor loading, noise limits and whether anyone on site can service the hardware are not secondary considerations; they decide what can be installed at all. Supermicro's announcement makes this concrete for its own rack by pairing it with in-row cooling distribution units that it rates at 1.8 MW each and deploys with redundancy. At the scale most private deployments start from, the same constraint appears as a duller question: how much power and cooling headroom the room has today, and what a second node would cost in both.
Serviceability matters as well. The warranty route, spare parts, on-site access and whether one node can leave service without losing the whole capability are operating requirements, and they belong in the brief.
What belongs in the sizing brief
A useful sizing brief is short and specific. Put in it: the model and the precision you intend to run; the context length one request may carry; how many requests may be in flight at once; the response-time target for the first token and for the completed answer; the availability you need and the hours during which it matters; the growth you expect over the next year; the power, cooling and space the room can supply; and the update and recovery expectations you have for the service.
With those answers, the hardware conversation becomes an engineering one. Without them, any server can be made to look suitable, because the claim is never tested against a requirement.
What an announcement does and does not establish
Two kinds of statement get confused. An announcement is a vendor's public position that a product exists, and sometimes that it is shipping. Availability is your supplier's ability to deliver that configuration, on a date, with the warranty and support terms you agreed. For Vera Rubin, NVIDIA's own release of 31 May 2026 said production shipments were set to begin starting this fall. Supermicro's statement of 23 September is a systems vendor saying it is shipping integrated racks now. Both are dated, public statements from their publishers, and neither is a quotation for a specific order.
That distinction is worth carrying into any hardware evaluation. An announcement can change what is worth investigating. It cannot tell you what is orderable, at what lead time, or how your workload behaves on it.
Where to start
Start from the workload, and write the requirements down before comparing anything. Bring the model, the precision, the context length, the concurrency and the response-time target to a sizing conversation, and treat every hardware claim as something to check against those figures rather than a reason to skip them.
If you are evaluating private infrastructure for inference, that requirement list is the right first agenda. LLM Machines sizes, configures and maintains private inference environments on infrastructure our customers control, and a sizing conversation starts with the workload rather than the bill of materials.