2,140 milliseconds for an arrival time
GET /parcels/7731/eta 200 2140ms. The tracking page of a parcel-delivery company shows nothing until that call returns. Behind it, the tracking service asks the ETA model for an arrival window and waits, with no timeout, while the model's few workers get through the requests queued ahead. At 09:00 the delivery notifications go out, a million people open their tracking links, and each of them waits on that queue.
The model itself is fine. Model serving is everything between a trained model file and a prediction a product can rely on: loading it into processes, putting it behind an endpoint, feeding it inputs, and keeping it fast, versioned and watched. Each run of the model on an input is an inference. Whether you run your own ML Inference service or call a hosted model, the product treats it like any remote dependency, with timeouts and rate limits.
Does anyone have to wait for it?
Most of the company's predictions have no customer waiting. Every evening the depots plan tomorrow's routes from an estimate for every parcel on every van. That is batch inference: a scheduled job runs the model over a whole table and writes the predictions where the planner reads them. Nobody waits on any single row, and a failure means a rerun.
The tracking page needs the van's last GPS ping, so it uses online inference: one request, one answer, inside the page's time budget. Each prediction moved to batch is one less model call that can time out in front of a customer, and a common design does both: batch the whole table overnight, go online only for what changed since.
Fitting inside the page
The tracking API has a 300 ms budget at the 99th percentile, because the slowest requests are the ones customers remember (Latency percentiles & tail latency). The ETA call gets a 120 ms timeout, and when it expires the page falls back to last night's batch estimate, less precise but already there.
The model's inputs, such as the van's last position and the stops left, are read by key at request time from a feature store that serves them under the same definitions training used: Feature stores: online, offline & training-serving skew. Forty people refreshing one parcel within a minute can share one inference while the model version and the inputs it read stay the same, cached under a key built from the parcel id, the model version and those inputs (Caching layers); requests that miss at the same moment can each run it, the stampede covered in Hot keys & cache stampedes.
Many inputs, one pass
Under the morning peak, the problem becomes throughput. A neural model on a GPU often takes little more time for 32 inputs than for one, up to a point, so one request per call leaves most of the device idle. Dynamic batching collects the requests that arrive within a short window, a few milliseconds, or up to a maximum batch size, and runs them as one batch; inference servers such as NVIDIA Triton offer it as configuration. The price comes out of the budget: a request may wait up to the window before its batch starts. A small tree model on CPUs gains little from it.
An autoscaler adds replicas on requests in flight or queue depth, since CPU usage says little about a busy GPU. What is particular to models is start-up: a new replica must load the model before it answers, seconds for a small model and minutes for a large one, so the service keeps a warm minimum and scales ahead of the peak it knows is coming. Sizing GPU pools belongs to GPU & accelerated compute.
Which model answered?
The file comes out of a training side (a Dataset Store, a Labeling Tool, a Training Job) into a Model Registry, which versions each artifact so serving can promote or roll back a specific one. Drawing those four is not enough on its own: serving must pin a specific registry version, eta version 14, not whatever is newest. Otherwise nobody can say which model produced last Tuesday's estimates, and there is no known-good version to go back to. On Kubernetes the serving box is often KServe; on serverless GPUs such as Modal it is a function you never provision. A hosted model needs the same pin, a model name with its version; large language models bring problems of their own, covered in Serving LLMs: gateways, fallback & token budgets.
Version 15 meets real traffic
Version 15, trained on a winter of data, beats version 14 on last month's deliveries, which is a claim about the past. Before it answers customers, it can answer real requests in shadow:
- The tracking service sends each ETA request to version 14 and returns version 14's answer to the page.
- The tracking service also sends a copy of the request to version 15 and does not wait for the reply.
- Both versions log their predictions under the request id; nobody sees version 15's.
- When the parcels arrive, a nightly job joins both versions' predictions to the actual delivery times.
Both versions load a pinned artifact from the registry. The page only ever gets version 14's answer; version 15's predictions go to the monitor to be compared.
Shadow traffic shows version 15 real inputs at real volume with no customer exposed. It finds what offline evaluation cannot, such as an input missing online or 400 ms of latency on the real request mix. It costs a second copy of the inference capacity, and a model whose output triggers an action, such as a notification, must have that action switched off in shadow.
Shadow cannot show how customers react, so next comes a canary: version 15 answers for real for 5% of parcels, split by parcel so a customer refreshing the page sees one model, then 25%, then all. Weighted routing and automatic rollback are in Canary Deployment. For a model, the comparison must include prediction quality, not only errors and latency, because a wrong model answers quickly with a 200. Quality also waits for labels, an ETA's error being known only when the parcel arrives, so each step lasts at least that long. Whether version 15 is better for the business is a separate question, for A/B testing & experimentation.
Rollback is pointing serving back at version 14, which is quick only if version 14 can still be loaded, so keep the previous version deployable until the new one has lived through a full cycle of labels.
Fast, healthy and wrong
Then a new depot opens in the north, and its vans use a ring road the training data has never seen. The dashboards stay green at 85 ms at the 99th percentile, while the estimates for that depot run 40 minutes optimistic. A model can be fast, available and wrong at once, and a latency dashboard cannot tell.
A Model Monitor (or Arize, Evidently) watches what latency cannot. Input drift compares the live inputs with the training data, such as the share of stops in postcodes the model never saw; prediction drift watches the outputs, such as the average estimate per depot. Both show up without labels. When the actual delivery times arrive, the monitor computes the real error per segment. Inputs can shift without hurting accuracy, so drift starts an investigation, not a rollback. When it does hurt, the usual answer is to retrain on recent data and send the new version through shadow and canary again.
Give version 15 a way in
The serving half of the ML Serving & Feature Store challenge asks for an autoscaled inference service, versions in a Model Registry, a split of live traffic between two versions with their metrics compared, and prediction latency and input drift watched against the training data. Pin the service to one registry version, send the new version through a shadow copy before the split, and put Prometheus on prediction latency per version. On that Grid, a rollback should be one pin changing, with the old version still there. Real-Time Fraud Detection and Content Moderation at Scale set the same problem for models whose mistakes cost more than a late parcel.