Every European company that looks at AI runs into the same question within the first meeting: where does our data go? In Germany, 66% of companies already using AI name data protection requirements as a barrier, 56% legal uncertainty and 45% the risk of data leaks (Bitkom). Across Europe, Deloitte found that 77% of companies take the country of origin of their AI vendor into account, and 66% are concerned about dependence on foreign AI (DEV Community, Deloitte data).
The reflex answer is “let’s run our own model on-prem”. Sometimes that is exactly right. Often it is an expensive way to get a weaker model plus the same compliance paperwork. This guide maps the full range of options between “public chatbot” and “air-gapped server room”, explains when each one makes sense and what drives the cost, and gives you a rollout plan that does not start with buying hardware you may not need.
“Private LLM” is a spectrum, not a product
| Level | Where processing happens | Model quality | Operational effort | Typical fit |
|---|---|---|---|---|
| 0. Consumer AI apps (“shadow AI”) | Provider’s consumer service, unclear terms | High | None | Never for company data |
| 1. Enterprise API with a DPA | Provider cloud, often US; contractual no-training; EU residency where offered | Frontier | Low | Most business tasks with non-sensitive or pseudonymised data |
| 2. EU-hosted models | EU provider or EU region of a cloud | High to frontier | Low | Personal data, EU-only customer contracts |
| 3. Open-weight model, self-hosted in an EU data centre | Your dedicated or private-cloud GPU servers | Good to high, model-dependent | Medium to high | Confidential data, steady volumes, sovereignty requirements |
| 4. Fully on-prem or air-gapped | Your own premises | Good, limited by hardware | High | Critical infrastructure, strict trade secrets, sites without internet |
Two things stand out. First, levels 1 and 2 already answer most GDPR questions when the contracts are done right: a data processing agreement, a subprocessor list, no training on your data and EU hosting where needed. Second, the step from level 2 to level 3 is not a compliance step but an operations step. From that moment you are responsible for uptime, security patches, model updates and capacity.
When self-hosting is the right call
Self-hosting (levels 3 and 4) pays off when at least one of these is true:
- Contracts or regulation exclude third parties. Customer contracts that forbid processing outside your infrastructure, professional secrecy in law firms, medical practices or tax advisory, public-sector tenders that require sovereign hosting, or R&D and M&A documents you would not send to any vendor.
- Volume is high and steady. Tens of thousands of documents per day or constant internal assistant traffic keep GPUs busy. Idle GPUs are the most expensive way to run AI.
- You need control over versions. Cloud providers update and retire models on their own schedule. A self-hosted model changes only when you decide — valuable for validated workflows.
- Connectivity is limited. Production sites, warehouses, ships or field locations with unreliable internet.
Self-hosting is usually the wrong call when volumes are low or spiky, when the task needs the strongest available reasoning — complex analysis, long multi-step agents — or when nobody in-house can own a GPU service around the clock.
In practice, most European companies we work with land on a hybrid: frontier cloud models for general tasks and an open-weight model in an EU data centre for the sensitive slice. That matches the broader market pattern — OpenAI, Anthropic or Google in the cloud for everyday work, Llama or Mistral on own servers for sensitive tasks (HouseofMVPs, ment.tech).
A typical scenario
Take an accounting firm with 40 employees. For draft emails, summaries of professional articles and internal questions about procedures, an enterprise API with a DPA and EU hosting is enough — no client secrets are involved. For analysing client financial statements, tax filings and payroll data, an open-weight model runs on a rented GPU server in a German or Dutch data centre. A gateway recognises from the data source which route is allowed. The result: staff work in one interface, the firm keeps control over what must not leave the building, and GPU costs arise only for the part that truly needs them.
The legal map on one page (not legal advice)
Self-hosting changes where your data goes. It does not switch off your obligations.
- GDPR. You still need a legal basis, purpose limitation, retention rules and access control. With external providers you add a data processing agreement under Article 28 and a subprocessor list. High-risk processing — call recordings, voice biometrics, health data — needs a data protection impact assessment (DPIA). Minimising transfers to the US is both a legal and a commercial argument.
- EU AI Act, transparency. Article 50 applies since 2 August 2026: people must be told when they are dealing with an AI system, and synthetic content must be labelled (AI Act, Article 50).
- EU AI Act, high risk. After the Digital Omnibus (Regulation 2026/1744), high-risk systems under Annex III — hiring, credit scoring, education, essential services — follow from 2 December 2027 (ActuIA). These duties attach to the AI system and its use, regardless of where the model runs.
- Model providers. Obligations for providers of general-purpose AI models apply since 2 August 2025. If you simply run an existing open-weight model, they mainly sit with its publisher; substantial modification, such as heavy fine-tuning, deserves a closer legal look.
- Fines. Up to €35 million or 7% of global turnover for prohibited practices and up to €15 million or 3% for most other breaches. For SMEs, the lower of the two amounts applies (ActuIA).
- National supervision. In Germany, the Bundesnetzagentur has supervised AI under the KI-MIG act since 29 July 2026.
For deadlines, duties and a practical checklist, read our guide to the EU AI Act for small and mid-sized businesses.
Choosing a model you can host
Open-weight models — Llama, Mistral, Qwen, DeepSeek and OpenAI’s gpt-oss among them — can be downloaded and run on your own hardware. Mistral, Aleph Alpha and LightOn are the European names that buyers most often shortlist for sovereignty reasons (DEV Community, Deloitte data).
Three rules from practice:
- Test in your language, on your documents. Public leaderboards are mostly English. A model that shines there can stumble over German compound nouns, Dutch legal language or your internal product codes. Build a small eval set from real tasks first — see how to set up AI evals.
- Read the licence. “Open weights” does not always mean “use it however you like”. Licences differ on commercial use, user thresholds and attribution.
- Size the model for the task, not for the headline. A mid-sized model with good retrieval often beats a huge model with poor retrieval. Smaller models also need fewer GPUs and answer faster.
Architecture of a private AI stack
A production-grade private setup has more parts than “a model on a server”:
- Inference server on one or more GPUs, serving the open-weight model through an OpenAI-compatible API, so applications do not care where the model runs.
- LLM gateway in front of all models, routing each request by data class: personal or confidential data to the private model, everything else to cloud models where allowed. Details in our LLM gateway guide.
- Retrieval (RAG) with permissions. A vector database such as Qdrant plus document ingestion that respects the access rights of the source systems. Without that, your assistant will cheerfully show the salary spreadsheet to the whole company.
- Identity and access. Single sign-on, roles, and per-user permissions passed through to tools and documents.
- Logging, monitoring and evals. The same discipline as with cloud models, with logs stored inside your perimeter.
- Redaction layer for hybrid setups: personal data is removed or pseudonymised before any request leaves your network.
Integrations are usually the expensive part. In RAG projects, connecting source systems accounts for 40–60% of the budget (ment.tech) — and that share does not shrink when the model moves in-house.
What it costs — and what drives the cost
There is no honest single price for “an on-prem LLM”, but there are good reference points. For RAG assistants in Europe, published ranges are €7–14k for a starter setup with one to three sources plus €180–380 per month to run; €14–28k for a business setup with five to ten sources and CRM integration; and €28–45k for enterprise setups with multi-tenancy and compliance. Tokens and hosting add €180–700 per month (LSI Analytics). In published price lists of studios that offer closed-perimeter deployments, the on-prem option adds around 20% to the project price (NeuralOps). Ongoing operations in the mid-market typically run at $2–8k per month in retainers (Lets-Viz).
| Cost driver | Cloud API (levels 1–2) | Self-hosted (levels 3–4) |
|---|---|---|
| Model usage | Pay per token; scales down to zero | Fixed GPU cost (rented or bought), used or not |
| Setup | Integrations, prompts, retrieval | The same, plus inference stack, hardening and capacity planning |
| Operations | Monitoring, evals, prompt updates | The same, plus OS and driver patches, model upgrades, scaling, on-call |
| Model upgrades | Provider ships them — and retires old ones | You choose when, and you do the work |
| Compliance work | DPA, subprocessors, transfer assessment | Fewer external processors; internal controls still required |
The break-even logic is simple. Compare your expected monthly API bill for the sensitive workload with the fixed monthly cost of GPU capacity plus the people-time needed to run it. If the GPUs would sit idle most of the day, stay with cloud models and EU hosting. If they would be busy — or if contracts forbid external processing anyway — self-hosting wins.
From decision to production in six steps
- Classify data and use cases. Public, internal, confidential, strictly confidential — and which AI use case touches which class. One workshop and a spreadsheet are enough to start.
- Write a processing policy per class. For example: public and internal data may go to an enterprise API with a DPA; confidential data only to EU-hosted or self-hosted models; strictly confidential data only on-prem.
- Run a model bake-off. Two or three candidate models, your eval set, your languages. Compare quality, latency and cost side by side.
- Build the hybrid gateway. One entry point that enforces the policy automatically, so no developer can send confidential data to the wrong model by accident.
- Harden. Network isolation, secrets management, access reviews, backups, logging inside the perimeter and a DPIA where required.
- Operate. Monitoring, monthly eval runs, planned model upgrades and capacity reviews — the routine we describe in AI Ops after launch.
A quick decision guide
| Your situation | Recommended starting point |
|---|---|
| SMB, internal assistant, no special contracts | Enterprise API with DPA, EU hosting where available |
| Customer service with personal data, EU customers | EU-hosted models or EU region, redaction, strict log retention |
| Law firm, tax adviser, clinic | Hybrid: private model for client files, cloud models for generic tasks |
| Manufacturer with R&D documents and steady usage | Self-hosted open-weight model in an EU data centre plus permission-aware RAG |
| Critical infrastructure, sites without internet | Fully on-prem, smaller optimised models, offline update process |
Typical mistakes
- Buying GPUs before running evals. Hardware is the last decision, not the first.
- Assuming self-hosted means compliant. You still need access control, retention rules, logging, a DPIA where relevant and AI Act transparency.
- Underestimating operations. Drivers, security patches, model upgrades and capacity planning do not run themselves.
- Choosing a model on English benchmarks when your users write German, Dutch or Russian.
- Ignoring the licence of an open-weight model.
- RAG without permissions, which turns a private model into an internal data leak.
- No exit plan. Whatever you choose, keep prompts, eval sets and configuration in your own repositories, so you can switch providers or models later.
Conclusion
“Private AI” is not a yes-or-no decision. For most European companies, the right setup is a hybrid: well-contracted cloud models for everyday work and a self-hosted open-weight model only where contracts, secrecy or volume justify it — all behind one gateway that enforces the rules automatically. Decide by data class, prove quality with evals in your own language, and only then size the hardware.
Our Systems team designs and runs private and hybrid AI stacks in EU data centres, and our Build team integrates them into your applications. Book a free 15-minute audit and we will map your data classes to the right hosting option.