Most AI projects are planned up to launch day. Then the real work begins — and nobody planned for it. Prices change in the knowledge base, the model provider ships an update, customers start asking about a new product, a CRM field gets renamed, and the agent that was brilliant in week one quietly gets worse.
Market data shows how often this goes wrong. Reviews of AI projects in Russian-speaking markets describe a familiar pattern: after six months the value is invisible, employees are irritated, customers complain and the budget is overrun (CNews). Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027 (Gartner). Companies that treat AI as an operated system invest in exactly the right things: in KPMG’s Q3 2026 pulse, 70% of large US companies use dashboards, monitoring and quality evaluation for their agents, and 43% run token budgets (KPMG).
AI Ops is the discipline that keeps AI systems accurate, affordable, safe and compliant after launch. Here is what it covers, what to monitor, what it costs and how to set it up.
Why AI systems decay after launch
1. Knowledge drift
Prices, policies, opening hours and product ranges change; the knowledge base does not update itself. A stale knowledge base is the most common cause of hallucinations months after launch — and once customers catch the system being wrong, trust drops fast.
2. Model drift
Providers update models, change default versions and retire old ones on their own schedule. The same prompt can suddenly produce longer answers, different tool use or a new tone. If your system points to a “latest” alias, it can change without a single line of your own code changing.
3. Input drift
New products, a marketing campaign, a seasonal peak or a new market bring questions the system has never seen. Coverage that was 80% in month one can be 60% in month six without anything breaking technically.
4. Integration breakage
CRM fields get renamed, API versions expire, OAuth tokens lapse, a webhook changes its format. Agents tend to fail silently here: they still answer politely while no longer creating the ticket or the booking.
5. Cost drift
Usage grows, conversations get longer, someone adds a bigger model “temporarily”. Context growth alone can make long voice sessions far more expensive per minute than short ones — in one measurement up to $0.146 per minute against a typical $0.05–0.08 (HackerNoon).
6. Channel and infrastructure issues
Voice adds its own failure modes: latency spikes on shared infrastructure; concurrency limits — Retell, for example, includes 20 concurrent calls and charges for each additional line (Retell); and in the US, carriers labelling unbranded business numbers as “Spam Likely”.
Early warning signs
You rarely get a single dramatic failure. Instead, watch for small signals: the escalation rate creeps up a few points a month; staff start double-checking the agent’s work “just in case”; customers write “I already told your bot”; the monthly invoice grows faster than the number of conversations. Each of these is cheap to fix when caught early and expensive when it surfaces as a complaint from your largest customer.
The five pillars of AI Ops
| Pillar | What it covers | Typical tools | Rhythm |
|---|---|---|---|
| Monitoring | Uptime, errors, latency, integration health, failed tool calls | Tracing (Langfuse, Helicone), Sentry, uptime checks | Real time, with alerts |
| Quality | Online eval scores, hallucination flags, escalation and repeat-contact rates | Eval suite, LLM judge, human review | Daily to weekly |
| Cost | Cost per outcome, token use, cache hit rate, budget burn | LLM gateway, provider dashboards | Weekly, monthly report |
| Change management | Prompts, models, tools and knowledge as versioned releases | Git, CI eval gates, canary releases | Every change |
| Incidents and compliance | Runbooks, kill switch, post-mortems, AI disclosure, DPA, access reviews | Runbook, on-call rota, compliance checklist | On event; quarterly review |
What to monitor: a practical metric set
Business outcomes — the numbers management cares about:
- resolution or automation rate, booked appointments, processed documents;
- escalation rate to humans, and the reasons behind it;
- repeat contacts within 48 hours — a resolved ticket that comes back was not resolved.
Quality:
- online eval scores on a daily or weekly sample of real conversations (see our guide to AI evals);
- groundedness and hallucination flags, policy violations;
- user feedback and manual corrections in downstream tools.
Technical:
- error and timeout rates per provider and per tool;
- p95 latency — for voice, the target is roughly one second per turn, because people normally leave only about 200 milliseconds between turns (PNAS);
- fallback rate, which reveals provider problems before users notice them.
Cost:
- cost per outcome — per resolved ticket, booking or document;
- tokens per conversation and cache hit rate;
- budget burn against monthly limits, ideally enforced by an LLM gateway.
Security:
- blocked injection attempts, unusual tool calls, bulk data reads, new outbound destinations (see prompt injection defenses).
Manual QA in contact centres typically reviews only 1–2% of calls (AssemblyAI). Automated scoring makes it realistic to look at every conversation and send only the flagged ones to people.
Change management: treat AI changes as releases
Most AI incidents are self-inflicted: a prompt edit on Friday afternoon, a batch of new documents imported without review, a model switched “because it is cheaper”. The fix is boring and effective:
- Version everything: prompts, tool definitions, retrieval settings, model IDs pinned to exact versions, knowledge base snapshots.
- Gate every change with evals. No release if critical metrics drop.
- Roll out gradually. Shadow mode — the new version answers in parallel and is only logged — or a canary on 5–10% of traffic before the full switch.
- Keep rollback one click away.
- Plan model upgrades. When a provider announces a new version or a retirement date, run the full eval suite, compare cost and latency, and schedule the switch. Do not wait for the deadline.
- Give the knowledge base an owner and a calendar. Someone on the business side is responsible for prices, policies and product data; changes reach the AI on a fixed schedule, with a quick check.
Incident response: when the AI gets it wrong
Sooner or later, the system will tell a customer something wrong. What matters is how fast you notice, contain and learn:
- Severity levels. A clumsy sentence is not the same as a wrong price promise or another customer’s data in an answer.
- Kill switch. A way to disable actions — refunds, bookings, emails — while keeping informational answers running, or to route everything to humans.
- Human handover. Clear escalation when confidence is low or the customer asks for a person.
- Post-mortem. What happened, why the evals did not catch it, and which new test case now guards against it.
- Legal checks. If personal data was exposed, GDPR breach procedures apply. In the EU, also confirm that the AI disclosure required by Article 50 of the AI Act was in place.
The AI Ops calendar
| Frequency | Tasks |
|---|---|
| Daily | Alerts, flagged conversations, failed tool calls, budget burn |
| Weekly | Review of escalations and low-scored conversations; knowledge base changes; quick eval run |
| Monthly | Full eval suite; cost and quality report with cost per outcome; improvement backlog; access review |
| Quarterly | Model review — new versions, prices, retirements; routing update; compliance check covering AI disclosure, DPA, subprocessor list and retention; disaster-recovery test |
Taking over an AI system that is already live: the first 30 days
Many companies come to AI Ops with a system that is already in production — built by a former agency, an internal hackathon team or a vendor that has moved on. A practical 30-day takeover plan:
- Week 1 — inventory and access. Collect every prompt, workflow, integration, API key and provider account. Move keys into a secrets manager and make sure provider accounts belong to your company, not to a freelancer’s personal email. Document which model versions are actually in use.
- Week 2 — baseline. Switch on tracing and cost logging if they are missing. Pull a sample of 100–200 recent conversations or documents and score them by hand or with a calibrated judge. This is your honest “before” picture, often sobering.
- Week 3 — safety net. Build a first eval set from that sample, including the failures you found. Add budget alerts, a kill switch for actions and a clear human handover. Pin model versions.
- Week 4 — quick wins and roadmap. Fix the most frequent failure causes — usually outdated knowledge, a broken integration or a missing escalation rule. Then agree on a monthly rhythm, an owner on each side and the metrics that will appear in every report.
After 30 days you know what the system really does, what it costs and where it fails — the starting point every improvement needs.
What AI Ops costs — and who should do it
Market reference points for ongoing AI operations:
| Model | Typical price | Source |
|---|---|---|
| Retainer, one system, small business | $500–1,500 per month | Lets-Viz |
| Retainer, mid-market | $2,000–8,000 per month | Lets-Viz |
| Common tiered offer | Starter $3,000 / Growth $7,500 / Enterprise $15,000 per month | Digital Applied |
| Risk premium for model drift and token spikes | +15–20% | Digital Applied |
| Tokens | Included with a 20–30% buffer, re-invoiced with a 15–25% markup, or paid directly by the client | Digital Applied |
| Typical model spend of one agent at scale | $50–500 per month | Lets-Viz |
For comparison, a senior AI developer on staff costs around $8,300 per month fully loaded (Lets-Viz) — and one person cannot cover on-call, evals, integrations and compliance alone. For classic software, full maintenance is often budgeted at 15–30% of the build cost per year (BPA Develop). AI systems sit at the upper end, because quality has to be re-measured continuously, not just kept running.
Who should own it? Usually a split. Your team owns the business side: knowledge base content, policies, handling of escalations and the final say on changes. An internal platform team or an external partner owns monitoring, evals, model updates, cost control and incidents. Whatever the split, put it in writing: response times, what counts as an incident, which metrics are reported, and — if you pay per outcome — what exactly counts as “resolved”. Analysts at HFS expect that definition to become the next big dispute in service-level agreements (HFS Research).
Typical mistakes
- No owner on the business side, so the knowledge base starts ageing on day one.
- Monitoring uptime only. The system is “green” while its answers get worse.
- Pointing to “latest” model aliases without running evals on each change.
- Fixing incidents without adding test cases, so the same failure comes back.
- No budget alerts until the invoice arrives.
- Contracts without SLA definitions for response times and outcomes.
- Treating AI Ops as optional and cutting it first when budgets tighten — exactly when drift starts to bite.
Conclusion
An AI system is closer to a new team member than to a piece of software: it needs onboarding, supervision, regular feedback and updates whenever the business changes. AI Ops turns that into routine — monitoring, evals, cost control, disciplined releases and a calm incident process. Budget for it from the start, give it clear owners, and your AI will be better in month twelve than in month one instead of quietly worse. If your project never got that far, read why AI pilots stall and how to reach production.
Our Systems team runs AI Ops for support, voice and document agents — monitoring, monthly evals, cost reports and model upgrades, with your data in your own accounts. Book a free 15-minute audit and we will check the health of your AI system and show you the three things to fix first.