LLM Machines · Part 2 of 2
Extracting a purchase-order number from a short email and comparing two hundred pages of technical specifications are different jobs. Even when both use the same AI model, the amount of input, the expected output and the acceptable waiting time can differ considerably. We need to understand those requests before recommending the hardware that will handle them.
LLM Machines configures and maintains the environment that runs the models, inside infrastructure the customer controls. The application sends requests through a documented inference API. Our work starts with the intended use of that service and continues as its software and usage change.
The first useful material is a small set of representative requests, using examples the business permits us to assess. For document extraction, that might include a clean document, one with missing fields and one with an unusual layout. The customer defines which results are acceptable and how errors should be handled. Those examples give us a basis for evaluating a proposed model and checking it again after a change.
Usage is the next part of the specification. A single person testing a model says little about the demand created by a department. We need an estimate of simultaneous requests, typical input and output lengths, and the busy periods that the system must accommodate. The application matters here too: a background job may tolerate waiting that would frustrate someone using an interactive tool. NVIDIA's benchmarking guidance distinguishes the time before the first output appears from the time between subsequent output tokens, helping describe performance in terms of what the application actually needs. NVIDIA's inference-cost benchmarking guide.
From there, we assess model suitability and the memory and compute needed to serve the expected work, together with storage, networking and the customer's availability requirements. Where new hardware is required, we coordinate procurement through a distributor after the customer approves the design. That puts a set of workload assumptions behind the hardware order and the maintenance quote.
An inference engine is part of that configuration. We use SGLang in our inference work. It provides model serving and mechanisms such as prefix caching and execution across multiple GPUs. Those capabilities create options for a deployment; their usefulness depends on the model, hardware and request pattern. A feature listed in the project documentation still needs an evaluation against the configuration we intend to support. SGLang documentation.
Alongside model serving, we configure which applications can call which models, how their credentials are managed and what usage limits apply. The API documentation gives the application owner a defined interface to work with. Compatibility still has to be checked for that application's requirements: ordinary text requests, a particular output format and tool-call behaviour each bring their own validation work.
We agree acceptance checks with the customer before the installation is handed over. The results need to identify the model and runtime versions and show how the service performed against the intended use. If the application must support a busy period, that load belongs in the validation. Availability and recovery commitments also need matching procedures and support coverage, with the required checks included in the delivery scope.
The handover documentation records the installed configuration, access arrangements, backup and recovery responsibilities, and the route for requesting a change or reporting a fault. A support engineer investigating a problem should be able to find those details without reconstructing the installation from someone's terminal history. The record has to remain useful after the original installer moves on to other work.
Updates create more of that work. In the wider serving ecosystem, vLLM's August 2026 release, version 0.28.0, changed its default token budget for batching from 8,192 to 16,384 and removed several deprecated configuration options. An operator using those defaults or options has concrete changes to review before upgrading. We use release notes to identify what needs examination, then evaluate a proposed change against the workload the environment is meant to support. vLLM 0.28.0 release notes.
Our ongoing service covers controlled updates, security maintenance and operational support within the agreed scope. Monitoring depends on the access the customer authorises. Where hardware telemetry is available, it can help establish whether a problem sits with the machine or the software. If that access is restricted, the support arrangement must account for the evidence the customer's team will need to collect. Monitoring the infrastructure does not require giving us routine access to the content of the business's documents.
Usage can change just as much as the software. An application that once sent a few paragraphs may begin submitting entire manuals, or another department may start using the same service. Capacity reviews examine that demand against the performance targets. Longer queues call for an examination of the requests and the serving configuration before deciding what to change or whether to add hardware.
The application remains the customer's or its software supplier's responsibility. That includes its permissions, stored documents, conversation history and any actions it performs in other systems. Our standard service covers the inference core and its API. If a business wants help connecting the service to a wider workflow, we can scope that separately, with a named owner, an outcome to measure and a handover.
A single server used during business hours and an environment requiring failover and extended support involve different operating work. We define maintenance around the infrastructure footprint, supported models and runtimes, update policy, recovery responsibilities and access arrangements. Pricing follows discovery because those choices determine what we will be responsible for maintaining.
For a first conversation, bring:
- A description of the job the application should do, with sanitised examples where possible.
- The result quality and waiting time the business can accept.
- An estimate of usage, including simultaneous requests and busy periods.
- The data restrictions, existing infrastructure and support-access constraints.
- A named owner for the application and its business requirements.
With that information, we can assess whether managed private inference fits the workload and define a configuration and service scope to evaluate. Tell us what you want to run.
Read the first article: Introducing LLM Machines: the case for private inference.