A service that answers its first request correctly has proved two things at once. It shows that the model, the hardware and the configuration can produce an answer. It also marks the moment a system begins to exist: a model someone can replace, a queue that can fill, a version that can change behaviour, and a machine that can stop. Most buying guides end at the first point. This one is about the second.
Inference, in this context, means running an AI model to produce an output. The earlier guides in this series covered how to shortlist a model, how an engine reuses repeated input, what to check in a server, and how to design an acceptance test. Each of those decisions leaves something behind to operate. An operating plan is the short set of agreements that says who does what once the service is live.
Two owners, not one
Start by naming an owner for the inference service and an owner for each application that uses it. They are usually different people with different responsibilities. The inference owner looks after the model in service, the serving configuration, the access rules and the update schedule. The application owner looks after the client or business system that calls the model, its permissions, its stored data and what it does with an answer.
The document that separates them is a documented inference API with an agreed connection test. Above that line, behaviour belongs to the application. A help-desk tool that summarises tickets is an application decision. Whether the model behind it is served at a particular precision and context length is an inference decision. Writing down which side owns which is what stops the two teams discovering the split during an incident.
What an access policy controls
An access policy sounds administrative, but it decides how far a mistake can travel. A credential issued to one application, with a request quota attached to it, means a runaway loop or a leaked key stops at that boundary instead of consuming the whole service. A policy that names which models a team may call keeps an experiment away from the production route.
The operating questions are concrete. Who can issue a credential, and who can revoke one. Which models each credential may reach. What happens when a quota is reached: does the request wait, or does it fail. Whether requests are logged, and who may read the logs. These are settled once, in writing, and then they apply to every later incident.
An update is a decision, and a default can change
Engine releases are not all repairs. Some change behaviour on purpose. SGLang's release of 2 October is a useful example: in that version the prefix cache began running on a Rust core by default, and a serving instance can now switch between prefill and decode without a restart. Both change how a service behaves after an upgrade, not only how fast it runs.
That is why an operating plan names who accepts a new version, on what evidence, and how it would be reversed. The acceptance test from the earlier guide supplies the evidence: the workload, the response-time target, the quality threshold, run twice. The rollback is the other half, and it is easier to agree before the window opens than after a metric moves. A version accepted once is not accepted in general. The next release is a new decision.
Recovery is a number before it is a diagram
Recovery planning often starts with a picture of what could fail. It is more useful to start with a time: how long until the service can answer again, and what that number is allowed to be. For a large model, most of that time is spent loading the weights. In a measurement the SGLang and Ant teams published on 21 August, weight loading accounted for about 94 percent of engine startup for one large model, and a persistent weight cache reduced the reload from roughly eight minutes to under a second of mapping. That is their configuration and their result; it shows where the time goes rather than promising a figure for any deployment.
The operating version is a recovery scope: what counts as usable again, whether an in-flight request survives, who can declare the recovery complete, and when the procedure was last exercised. A recovery plan that has never been run is a document. A drill that returns a time is evidence.
Monitoring only sees what you grant
Monitoring is limited by access. A service can report utilisation, queue depth and error rates, but only where someone can reach that information. If the customer keeps remote access closed, some checks are the customer's to run and others fall to whoever is on site. The plan should say which is which, and it should describe the arrangement the customer has approved rather than the one that makes the best dashboard.
Capacity belongs in the same section, with a date and an owner. Comparing current utilisation against the assumptions the system was sized for is how a changed workload becomes visible before users notice it. It is the one part of the plan that is about the future rather than the present.
Keep the integration outside the core
Business-workflow integration is a different kind of work from running inference, and it belongs in a separate scope. A model that answers questions is an inference service. A system that files a purchase order or sends a message has taken an action in a business process, and that action needs its own owner, its own acceptance criteria and its own evidence. Folding the two into one promise makes both harder to operate.
The test is the same boundary as before. The inference service offers a documented API and a smoke test. The application, and any workflow it drives, stays with the customer or its software vendor, unless a separate piece of work is agreed with its own measurable outcome.
The first morning something changes
Take a small example. A private inference service starts returning slower answers on a Monday morning. If a plan exists, three things are already settled: who can see the queue depth, who may change the serving configuration, and which version was accepted and how to return to the previous one. If none exists, the morning is spent working out whom to ask.
The earlier guides in this series covered what to buy and how to test it. What they leave behind is an operating question: who owns the model, who owns each application, who can change the service, and what you would do if it stopped. Those answers are worth writing down before you need them, and revising when the workload does.
If your team is weighing up what to run in-house, the workload and the controls you want to keep are the right place to start. We can work through the configuration and maintenance scope from there.