RAG Knowledge Assistant: Architecture, Costs and Pitfalls

Every company has the same problem in a different shape: the answer exists, but nobody can find it. The current price list is in a SharePoint folder, the exception policy in a Confluence page from 2024, the technical detail in a closed ticket and the real explanation in a colleague’s head. New employees ask the same questions; experienced ones answer them again and again.

A knowledge assistant built on retrieval-augmented generation (RAG) promises to fix this: ask in plain language, get an answer from your own documents with a link to the source. Demand is real — in Anthropic’s 2026 survey, 41% of technical leaders named knowledge-base management among the most effective uses of AI agents (Anthropic).

But the internal chatbot is also the AI project most likely to become shelfware. The demo impresses, then usage drops because answers are outdated, permissions leak or nobody can tell whether an answer is right. This guide explains the architecture that avoids those failures, what it costs in 2026 and how to roll it out.

How RAG works, in plain language

1. Ingest. Connectors pull documents from your systems and keep them in sync, including updates and deletions.
2. Parse and chunk. Documents are converted to clean text and split into passages that keep headings, tables and context together.
3. Index. Each passage is stored with an embedding (a numerical representation of meaning) and with keywords and metadata such as source, date, language and access rights.
4. Retrieve. When someone asks a question, the system finds the most relevant passages the user is allowed to see.
5. Rerank. A second, more precise model reorders the candidates so the best evidence comes first.
6. Generate. The language model writes an answer using only those passages and cites them.
7. Log and evaluate. Questions, retrieved sources, answers, feedback, latency and cost are recorded so quality can be measured.

The model is only step six. Most of the quality — and most of the work — sits in the other six.

Reference architecture

Permissions are the hard part

Integrations and access control typically consume 40–60% of a RAG project’s budget (ment.tech). The reason is simple: if the assistant can read HR files, board minutes or salary data, someone will eventually ask the question that surfaces them. Permissions must be enforced when passages are retrieved, based on the user’s real access in the source system. Telling the model “do not reveal confidential data” is not access control.

Retrieval quality beats model choice

Most wrong answers are not caused by a weak model but by the wrong passages. Proven improvements: hybrid search (semantic similarity plus exact keywords, which matter for product codes and names), reranking, metadata filters (country, product line, validity date) and chunking that keeps a table together with its heading.

Citations and “I don’t know”

Every answer should link to its sources so users can verify it in one click. Equally important: when retrieval finds nothing relevant, the assistant must say so and suggest who to ask, instead of improvising. An honest “I don’t know” builds more trust than a fluent guess.

Connectors via MCP

The Model Context Protocol has become the standard way to connect AI systems to company tools. Since December 2025 it has been governed by the Agentic AI Foundation under the Linux Foundation (MCP blog), and there are more than 10,000 public MCP servers (Digital Applied). That shortens connector work, but each server still needs a security review. Our article on MCP integrations explains how to connect them safely.

Use cases by team

The same architecture serves very different teams. What changes is the sources, the permission model and the definition of a good answer.

RAG or fine-tuning?

A common question in early conversations is whether to “train the model on our data” instead. For company knowledge, the answer is usually no. Fine-tuning changes how a model writes, classifies or follows a format; it does not reliably teach it facts, cannot cite a source and has to be repeated every time a document changes. Worse, it cannot respect permissions — whatever is trained into the model is available to everyone who uses it.

RAG keeps knowledge outside the model, current and attributable. Fine-tuning can still play a role later, for example to make a smaller, cheaper model follow your answer format, but it is an optimisation, not the foundation.

Buy or build?

Built-in assistants — Microsoft Copilot, assistants inside Confluence, Notion or your helpdesk — work well when almost all knowledge lives in one suite and its permission model. Microsoft Copilot Studio, for example, is priced at $200 per 25,000 credits (CloudZero).

A custom assistant makes sense when:

• knowledge is spread across several systems that no single vendor covers;
• permissions are complex, for example by country, client or project;
• you need several languages with consistent quality;
• data must stay in the EU or on your own servers;
• answers should lead to actions, such as creating a ticket or updating a CRM record;
• you need measurable accuracy per domain, not a general impression.

The basic “chat with our documents” bot is becoming a commodity. The value of a custom build lies in the parts a generic tool does not handle: your permissions, your sources, your quality measurement.

What it costs in 2026

Sources: LSI Analytics for European prices, ment.tech for US prices.

Running costs for tokens and hosting are typically €180–700 per month (LSI Analytics); at scale an average agent spends $50–500 per month on models (Lets-Viz). You keep them predictable by routing simple questions to cheaper models, caching frequent answers and setting budgets with alerts. See our article on the LLM gateway and cost control.

Measuring quality with evals

Before launch, collect 100–200 real questions from the teams that will use the assistant, with the correct answer and the source document confirmed by the content owner. This test set is your quality contract.

Re-run the set after every change to prompts, models, chunking or sources. Without it, every “improvement” is a guess. Our article on AI evals covers the method in detail.

Data protection and hosting in Europe

European buyers care where their AI comes from: according to Deloitte data, 77% of companies consider the AI vendor’s country of origin and 66% worry about dependence on foreign AI (DEV Community). In Germany, data protection requirements are the top barrier for 66% of companies already using AI (Bitkom).

You have three main options:

1. Cloud model with EU data processing and a DPA — the quickest start, with the strongest models.
2. EU-hosted model endpoints — the model runs in an EU region, with contractual guarantees on data use.
3. Self-hosted open-weight models (for example Mistral) on your own or EU servers — maximum control, more operational work.

Whatever you choose: do not index everything (HR records and personal data often do not belong in a general assistant), define retention for logs, and make sure deletions in the source system reach the index — otherwise the right to erasure under the GDPR becomes a problem. For the self-hosted route, see private and on-prem LLMs.

Common pitfalls

• Stale sources. If nobody owns the content, the assistant confidently repeats last year’s prices. Give every source an owner and a review date.
• Permission leaks. Retrieval without access filtering is a data breach waiting for the right question.
• Indexing everything. More documents do not mean better answers. Start with the sources people actually need and that are maintained.
• Broken tables and scans. Price lists and specifications often live in tables; poor parsing turns them into noise.
• No evaluation set. Without it, nobody can tell whether a change helped or hurt.
• Prompt injection via documents. An uploaded file or web page can contain instructions aimed at the model. Treat content as data, restrict tools and keep sensitive actions behind human approval.
• No adoption plan. Put the assistant where people work, train them in 30 minutes, and publish what it can and cannot answer.
• Lock-in. Keep the index, prompts and test sets in your own accounts so you can change the model or vendor.

A 6–8 week rollout

1. Weeks 1–2: scope. Choose one team and one domain (for example customer support or sales engineering), list the sources and owners, collect the test questions.
2. Weeks 2–4: build. Connectors, parsing, hybrid index, permissions, first answers with citations; measure against the test set.
3. Weeks 4–6: pilot. Ten to thirty real users, daily feedback, weekly evaluation runs, fixes to sources and chunking.
4. Weeks 6–8: launch and handover. Open to the whole team, set up monitoring, cost alerts and a monthly content review.

Conclusion

A knowledge assistant succeeds or fails on unglamorous details: which sources it reads, who keeps them current, whether permissions are enforced and whether quality is measured. Get those right and the model choice becomes a detail you can change later. Get them wrong and even the best model will produce confident nonsense.

If you want a realistic estimate for your sources, permissions and hosting requirements, book a free 15-minute audit. We will tell you whether a built-in assistant is enough or where a custom RAG setup pays off. Learn more about our AI agents and systems integration work.