Two providers, one code path
429 Too Many Requests, on every summary request a contract-review product sends to its model provider. Law firms upload the weekend's contracts first thing in the morning, and ten minutes into the rush the account's tokens-per-minute limit is spent. The team has an account with a second provider, opened after an outage in the spring. Nothing uses it: the summariser calls provider A's SDK directly, with A's model name and a prompt tuned for A, so switching takes a deploy.
The model is not the problem; the path to it is, with no place to decide anything per request. This article is about that path: the layer between an application and the models it calls, hosted or self-hosted, where each request gets its model, its prompt, its tenant's budget and a record of what it cost.
One door to every model
An LLM Gateway is a service every model call in the product goes through. The summariser sends it one request format naming a route, contract-summary, rather than a vendor's model; the gateway holds the provider credentials, translates to each provider's API and applies the rules no caller should copy: timeouts, retries, and which provider to try next. LiteLLM is a common open-source one.
The rules are those of any remote dependency. A 429 usually says when to come back, and an immediate retry only spends more of a limit already spent; Retry covers backoff and jitter. A provider that keeps failing leaves the rotation for a while, which is Circuit Breaker. Many teams also screen prompts and responses at the gateway, with a Guardrails step that checks them against a policy. When the caller is an agent and the model's answer becomes an action, that screening is not enough: checking the proposed tool call in code, and having a person approve it, is Agent guardrails: checking tool calls & human approval.
Falling back without breaking the answer
Now the contract-summary route reads: provider A first; on a 429, a 5xx or a timeout, provider B. The morning rush gives slower summaries instead of failed ones. Then the first fallback summaries reach customers, with three problems a list of providers does not solve:
- The prompt was written for A. B follows the same instructions differently: longer summaries, the parties named in another place, clause numbers dropped.
- The contract does not fit. A 300-page lease fits within A's context window and not B's, so the fallback fails on exactly the large documents that were slow to begin with, unless the route counts the tokens first and keeps those on A.
- Some firms never agreed to B. Their engagement terms list the providers allowed to process client documents, so routing must respect a list per tenant, and for those firms the only fallback is to wait.
So the route keeps a prompt per model, stored as versions in a Prompt Registry and pinned by the route instead of pasted into code: contract-summary version 14 for A, and a version 14 of its own for B. A change to either prompt, or to the model behind it, goes through an Eval Harness first. It runs a fixed dataset through the stack offline (in CI, on every prompt or model change), here a few hundred contracts with summaries the firms' lawyers accepted, and scores the new output against them. A fallback nobody evaluated is a second product nobody tested. What else to answer when a dependency fails, from a cached value to a clear error, is Fallback.
Tokens are the bill
In April a large firm uploads the data room for an acquisition: 4,000 documents in one morning. The 4,000 requests can stay within what the Rate Limiter in front of the API allows, since it counts requests, not their size. In tokens, at around 25,000 a document, it is 100 million, drawn from the same provider limit every other firm shares, and their summaries start waiting behind the data room.
An AI Cost Guard sits between the app and the LLM Gateway and enforces token budgets per tenant, a share of the provider's tokens per minute as well as a monthly spend: the Rate Limiter counts requests, this counts tokens and what they cost. Before a call it reserves an estimate, the input's token count plus the output cap the request sets, atomically so concurrent calls cannot spend the same budget twice; the gateway asks for a new reservation before each retry or fallback and reports what each attempt used, and the guard corrects the charge. Per-token pricing from hosted inference providers (OpenAI, Anthropic, or Fireworks AI for open-weight models) is what makes that budget the number that matters.
A tenant near its budget has its bulk work delayed while its interactive requests go on, the choice Throttling describes. A job like the data room can also go to a batch endpoint, which several hosted providers offer at a lower per-token price for results that come back within a day. A Semantic Cache, which saves calls when many users ask near-identical questions, helps little here: unique contracts rarely repeat, and a cache not partitioned by firm would leak one firm's documents to another.
Thirty seconds to the last word
A long contract's summary can take half a minute to generate, and a blank screen that long looks broken, so the product streams: the gateway passes tokens to the browser as the provider produces them, usually over server-sent events (Realtime transport: long-polling vs SSE vs WebSocket compares the options). On the gateway, timeouts then measure the time to the first token and the gaps between tokens; the final token count usually arrives with the end of the stream, which is when the cost guard corrects its charge; and fallback stays clean only until the first token, because after that a switch restarts a summary the reader is already reading.
Which prompt, which model, what it cost
Two weeks later a partner complains that summaries have started to leave out the termination clause. LLM Observability records each call with the tenant, prompt version, provider and model, tokens, cost, latency and whether a fallback fired, in a tool such as LangSmith or Braintrust, and here the traces of the summaries the partner flagged all carry prompt version 15 on provider A, live since the day the drop began.
Version 15 had passed the eval harness, whose contracts rarely had unusual termination clauses: offline evals catch only what their dataset covers. So an LLM Judge also grades a sample of live traffic against a rubric, and what it fails goes to Human Review, where a person confirms the failure before the team changes a prompt or a route over it (approving an agent's action before it runs is in Agent guardrails: checking tool calls & human approval). Without traces and evals a quality regression ships silently: every call returns 200, and the summaries are simply worse.
Running the model yourself
One firm's contract forbids sending its documents to any third-party model provider. For that tenant the route points at an open-weight model the team runs on its own GPUs behind vLLM (its traces, and any judge model that reads its documents, stay in-house too), an inference server that batches concurrent requests and exposes an OpenAI-compatible API, so the gateway treats it as one more provider. Ollama runs open-weight models on one machine, for development and offline tests rather than a fleet.
Provider A, then B on failure, for firms that allow both; the firm whose documents cannot leave goes only to the team's own vLLM pool. Every call is traced with its prompt version and cost.
Self-hosting trades a per-token bill for GPUs that cost money whether or not anyone is summarising, and for the work of keeping them busy: batching, autoscaling and the cold start of loading weights are in GPU & accelerated compute; pinning model versions and rolling out a new one are in Model serving: inference, versions & safe rollout. The gateway's part stays the same: the route, its prompts, the tenant's budget and the trace, whichever box answers.
A gateway in front of your own GPUs
The Self-Hosted LLM Inference Challenge asks for one gateway API with per-tenant quotas in front of all models, tokens streamed to clients and a semantic cache, over a GPU pool. Draw that gateway as the only way in, with its fallback edge to a hosted provider labelled with when it fires.
What to read next depends on what sits behind the summariser. If the product starts answering questions over a firm's whole archive, the prompt needs retrieved passages: The RAG stack: embed, retrieve, rerank. If the caller becomes an agent that acts on the model's answers, read Agent guardrails: checking tool calls & human approval first. If one agent becomes several, AI Agent Orchestration patterns.