Most companies start their AI journey with one API key pasted into one application. Twelve months later there are five apps, three model providers, keys in places nobody remembers, and a monthly invoice that nobody can explain line by line. When one provider has a bad afternoon, the support bot goes silent with it.
An LLM gateway fixes this at the architecture level. It is a single layer between your applications and every model you use. It decides which model handles which request, how much each team or customer may spend, what happens when a provider fails, and what gets logged. Done right, it turns AI spend from a surprise into a line item you can forecast — and turns switching models into a configuration change instead of a rewrite.
The pressure is real. In KPMG’s Q3 2026 pulse survey of large US companies, 74% have built cost checks into their AI approval process and 43% already run token budgets (KPMG). In Germany, 54% of companies that use AI name unpredictable costs as a barrier (Bitkom). And Gartner expects spending on AI platforms and models to grow by 63% in 2026, which makes budgets harder to predict, not easier (Gartner).
What an LLM gateway does (and what it does not)
Think of it as an API gateway that understands tokens. Your applications stop calling OpenAI, Anthropic, Google or a self-hosted model directly. They call one internal endpoint — usually OpenAI-compatible, so existing SDKs keep working — and the gateway takes over:
- Routing. It chooses the model per request based on task, customer tier, data sensitivity or latency budget.
- Keys and budgets. It issues virtual keys per app, team or customer, each with spend and rate limits. The real provider keys live only inside the gateway.
- Caching. It returns stored answers for repeated requests and makes use of providers’ prompt caching for long, static prompts.
- Reliability. Retries, timeouts, fallbacks to another provider or region, circuit breakers.
- Observability. Tokens, cost, latency, errors and metadata for every call, exported to your monitoring stack.
- Policy. Redaction of personal data, blocked models for certain data classes, maximum context sizes.
What a gateway is not: it is not an agent framework, not a vector database and not a magic cost cutter. It gives you the controls; the savings come from the decisions you make with them.
There are three common ways to get one. Open-source proxies such as LiteLLM run on your own infrastructure, which suits European companies that want logs to stay in the EU. Managed gateways such as Cloudflare AI Gateway or Portkey are quicker to start. Observability-first tools such as Helicone or Langfuse focus on tracing and analytics and are often combined with one of the first two. Many teams also write a thin in-house layer — perfectly reasonable, as long as it is treated as production software with tests and an owner.
Where the money leaks without a gateway
In the setups we audit, overspending rarely comes from one big mistake. It comes from five small leaks that nobody sees, because nobody measures them.
1. One model for everything
The strongest model gets hard-coded during the prototype “to be safe” and stays there. It then classifies emails, extracts invoice numbers and writes one-line summaries — jobs a light model does just as well.
2. Context that grows with every turn
Chat history and retrieved documents are resent on every turn, so long conversations get more expensive with each step. In a measurement of 4,000 OpenAI Realtime voice sessions, typical cost was $0.05–0.08 per minute, but context growth pushed some sessions up to $0.146 per minute (HackerNoon).
3. Retries and runaway agent loops
An agent that calls a tool, misreads the result and tries again can burn through hundreds of calls before anyone notices. Without a step limit and a per-request budget, the invoice is the first alert.
4. No attribution
If every request uses the same key, you know what you spent, but not which feature, customer or experiment spent it. Pricing an AI feature or a client retainer becomes guesswork.
5. Paying for useless work
Some spend produces nothing at all. In outbound voice campaigns, for example, misdetected answering machines can waste 15–20% of the budget (MTS Exolve). Similar waste hides in duplicate requests, abandoned sessions and test traffic hitting production models.
The good news: none of this requires a big platform. It requires one point that every request passes through and where it gets measured.
Model routing: the biggest lever
Routing is where most savings come from, because the price gaps between model tiers are large. On OpenAI’s official price list, the full realtime audio model costs $32 per million input tokens and $64 per million output tokens, while the mini version costs $10 and $20 — roughly a third (OpenAI). In some vendors’ price lists the gap between the lightest and the top tier is tenfold (GigaChat tariffs).
In our own calculation for a ten-turn support dialogue with roughly 3,000 tokens of retrieved context per turn, a pro-class model costs about an order of magnitude more per conversation than a light one. That is why the fixed per-conversation prices that many chatbot products sell only stay profitable with routing.
Five routing patterns
| Pattern | How it works | Good for | Watch out for |
|---|---|---|---|
| Static by task | Each task type has a fixed model: light for classification, strong for reasoning | Most companies; easy to audit | Re-check whenever models change |
| Cascade | Try the light model first; escalate on low confidence or failed validation | High-volume support and extraction | Needs a reliable confidence signal |
| By customer tier | Premium plans get stronger models | SaaS products with tiers | Keep cost per customer visible |
| By data sensitivity | Personal or confidential data goes only to EU-hosted or self-hosted models | European companies, regulated sectors | Classifying data reliably |
| By latency budget | Voice and live chat get fast models; batch jobs take slower, cheaper paths | Voice agents, real-time UX | Fast models may need tighter prompts |
Route on evidence, not on hope
A cheaper model that gives 5% more wrong answers can cost more than it saves — in refunds, escalations and lost trust. The rule is simple: a task only moves to a cheaper model after it passes the same eval set as the current one. If you do not have such a set yet, start there; our guide to AI evals shows how to build one in days, not months.
Cost control mechanics that actually work
Routing decides who does the work. These controls decide how much work is allowed.
- Virtual keys with budgets. One key per application, per team and — for SaaS — per customer. A soft limit with alerts at 50% and 80%, and a hard limit at 100% with defined behaviour: switch to a lighter model, queue the request or hand over to a human.
- Hard caps per request. Maximum input context, maximum output tokens and maximum agent steps. A runaway loop then costs cents, not hundreds of euros.
- Context hygiene. Summarise long histories, send fewer and shorter retrieved chunks, strip boilerplate from documents before indexing. This often saves more than any discount.
- Caching. Exact-match caching for repeated requests such as product questions or status texts. Provider prompt caching for long system prompts and static instructions, which are billed at a discount when reused. Use semantic caching — reusing answers to “similar” questions — only for non-personal content, because a near match can be a wrong match.
- Batch for non-urgent work. Nightly document processing, enrichment and report generation can go through providers’ batch interfaces, which are priced below real-time calls.
- Cost per outcome, not per token. The number that matters is cost per resolved ticket, processed invoice or booked appointment. A model that is twice as expensive but resolves more on the first attempt can be the cheaper choice.
How big is the bill in practice? Industry pricing analyses put model spend for a typical agent at scale at $50–500 per month (Lets-Viz). For many SMBs, the main value of a gateway is therefore not “saving 80%” but predictability: no surprise invoices, no runaway loops, and reliable numbers for pricing and budgeting.
Who pays for the tokens?
If an agency builds and runs your system, settle this in the contract. The market uses three models: tokens included in the fee with a 20–30% buffer, tokens re-invoiced with a 15–25% markup, or the client paying the provider directly. Some agencies add a 15–20% risk premium for model drift and token price spikes (Digital Applied). We prefer the third model — your own provider accounts, our gateway configuration, full visibility — because it also keeps you independent. More on that in our guide to owning your AI and avoiding lock-in.
Observability: what to log for every call
Margins in AI are thinner than in classic software — Bessemer puts gross margins of AI companies at 50–60%, against 80–90% for SaaS (SaaS Mag). Inference is a real cost of goods sold and deserves the same discipline as cloud spend.
A useful minimum per request:
| Field | Why you need it |
|---|---|
| Timestamp, environment, application, feature | Attribution and debugging |
| Customer or tenant ID (pseudonymised), user role | Cost per customer, access audits |
| Provider, model and exact model version | Explains behaviour changes after updates |
| Prompt template version, retrieval settings | Links quality regressions to specific changes |
| Input, output and cached tokens; cost | Budgets, pricing, cache efficiency |
| Time to first token, total latency | User experience, voice turn-taking |
| Status, error type, retries, fallback used | Reliability and provider comparison |
| Tool calls with arguments and results | Agent debugging and security reviews |
| Eval score or user feedback, where available | Quality trend over time |
For agents, log traces, not just single calls: one user request may trigger ten model calls and five tool calls, and you need to see them as one tree. Langfuse or Helicone for tracing, Sentry for errors and PostHog for product analytics is a common combination in production AI packages (HouseofMVPs).
Latency deserves its own dashboard. In LangChain’s State of Agent Engineering 2026, quality is the number one barrier to production (32%) and latency the number two (around 20%) (LangChain survey). For voice, the bar is brutal: people leave gaps of about 200 milliseconds between turns (PNAS), while speech-to-text, LLM and text-to-speech pipelines currently land at roughly one second (DevToolLab). Watch the 95th percentile, not the average.
Logs are personal data
Prompts and outputs often contain names, email addresses, order details or health information. Under GDPR that means a defined purpose, a retention period, access control and, ideally, redaction before storage. Keep logs in the EU when your users are in the EU, list every model provider as a subprocessor in your data processing agreement, and decide upfront who may read raw conversations — and document all of it in your records of processing.
Reliability: fallbacks, timeouts and graceful degradation
Providers have outages, rate limits and latency spikes. A gateway turns these from incidents into non-events:
- Timeouts per use case. A voice turn cannot wait ten seconds; a nightly batch job can.
- Retries with backoff for transient errors — but never infinite.
- A fallback chain. The same model in another region or through another provider, then a comparable model from a second vendor, then graceful degradation: a polite handover to a human or a callback promise.
- A circuit breaker. If a provider fails repeatedly, stop sending traffic for a few minutes instead of piling up timeouts.
One catch: different models behave differently with the same prompt. A fallback model that has never been tested is a hidden quality risk. Run your eval set against every model in the chain and keep prompt variants per model where needed.
Build, buy or self-host? A quick comparison
| Option | Best for | Pros | Cons |
|---|---|---|---|
| Direct SDK calls, no gateway | One prototype, one model | Zero setup | No budgets, no attribution, hard to switch |
| Self-hosted open-source proxy (e.g. LiteLLM) | EU companies, data control, many models | Logs stay with you; no per-request fee; flexible | You run, patch and scale it |
| Managed gateway (e.g. Cloudflare AI Gateway, Portkey) | Fast start, small teams | Quick setup, dashboards included | Data passes another vendor; check DPA and region |
| Thin in-house layer | Specific routing or compliance logic | Exactly what you need | Easy to under-engineer; needs tests and an owner |
For most mid-sized European companies we end up with a self-hosted open-source proxy in an EU data centre, a tracing tool next to it, and routing rules stored in Git. If some data must never leave your network, the same gateway can send those requests to a self-hosted model — see our guide to private and on-prem LLMs.
A rollout plan in six steps
- Inventory. List every place that calls a model: applications, scripts, automations in n8n or Make, vendor tools. Note the provider, model, key owner and last month’s spend.
- One entry point. Put all calls behind the gateway, rotate the old keys and remove provider keys from code and automation tools.
- Tag everything. Every request carries application, feature, customer and environment. Without tags, steps 4–6 are impossible.
- Baseline for two weeks. Cost per feature, cost per outcome, p95 latency, error and fallback rates. This is your “before” picture.
- Optimise with evals. Move tasks to lighter models only where the eval set confirms equal quality; add caching and context limits; set budgets and alerts.
- Run it. A monthly cost and quality report, a quarterly model review and alerts routed to a named owner. This is the core of AI Ops after launch.
For a small or mid-sized company, a focused team can usually complete steps 1–4 in one to two weeks. Step 5 is where the savings appear; step 6 is where they stay.
Typical mistakes
- The gateway becomes a single point of failure. Run it redundantly and monitor it like any other critical service.
- Logging everything forever. Full prompts with personal data and no retention policy is a GDPR problem waiting to happen.
- Switching to a cheaper model without evals. Savings on the invoice, losses in customer trust.
- Semantic caching of personalised answers. One customer’s order status served to another is a data breach.
- Budgets without owners. An alert that lands in a shared inbox is not an alert.
- Forgetting embeddings, speech and image calls. They often sit outside the “LLM” budget and grow quietly.
- Lock-in through the back door. Keep routing rules and configuration in your own repository, so you can replace the gateway product as easily as a model.
Conclusion
An LLM gateway is not glamorous, but it is the difference between AI as an experiment and AI as an operated system. It gives you one place to choose models, cap spend, survive provider outages and prove what each feature costs. Start with the inventory and tagging, measure for two weeks, then let evals — not intuition — decide where cheaper models are good enough.
If you want a second pair of eyes on your current setup, our Systems team designs and runs gateways, budgets and observability for companies in the EU and the US. Book a free 15-minute audit and we will show you where your token spend goes and what to fix first.