In April 2023, Samsung engineers inadvertently sent confidential source code and strategic meeting notes to ChatGPT. The incident made headlines worldwide and exposed a major blind spot: using a third-party-hosted LLM potentially means entrusting it with your most sensitive data. Two years later, according to the PwC EMEA Cloud Business Survey 2025, 46% of French companies cite AI as their main reason for adopting sovereign cloud, compared with a European average of 30%. The signal is clear: the question is how companies will adopt LLMs without losing control of their data.
This article details concrete architectures—on premises, dedicated VPC and sovereign cloud—that let you run open-source models such as Mistral or LLaMA on your own data without a single token passing through a US public cloud.
TL;DR: Open-source models (Mistral, LLaMA, Phi) have reached performance levels rivaling proprietary APIs. Deployed on premises or in a dedicated VPC with inference engines such as vLLM, they process sensitive data without external exposure. Break-even against cloud APIs is reached from 30 million tokens/day, or 1–4 months of operation.
Why Your Enterprise Data Should Never Pass Through a Public LLM
The Structural Risk of Proprietary APIs
When an employee submits a prompt to an API such as GPT-4 or Claude through its cloud interface, the data leaves the company's perimeter. The provider receives the complete text, whether a customer contract, financial report or source code extract. Even with contractual no-retention clauses, three risks remain.
The first is technical: data passes through shared infrastructure, potentially subject to foreign jurisdictions. The US CLOUD Act, for example, authorizes federal authorities to demand access to data stored by US companies, including on servers in Europe.
The second is operational: provider logs may retain request traces, even temporarily. A provider security incident can then expose all customers' data simultaneously.
The third is regulatory: in its 2025 recommendations, the CNIL reiterates that GDPR applies fully to personal data processing through AI systems, with no exception for the innovative nature of the technology.
The Sensitive Data Involved
Companies do not all handle the same critical data, but most underestimate how much flows through their teams' prompts.
| Data type | Concrete examples | Exposure risk |
|---|---|---|
| Intellectual property | Source code, patents in preparation, business algorithms | Loss of competitive advantage |
| Customer data | Contracts, purchase histories, correspondence | GDPR violation, CNIL sanctions |
| Financial data | Financial forecasts, M&A, valuations | Insider trading, economic espionage |
| HR data | Evaluations, compensation, disciplinary files | Employment tribunal disputes, privacy violations |
| Medical data | Patient records, clinical trials | Aggravated CNIL sanctions (sensitive data) |
A Tightening Regulatory Framework
GDPR is the foundation, but the European regulatory landscape is becoming more complex with the AI Act, progressively taking effect since 2024. Companies deploying high-risk AI systems in healthcare, HR and finance must now document data governance practices and demonstrate compliance. Using a proprietary API hosted outside the EU makes that demonstration significantly harder.
NIS2, applicable since October 2024, strengthens cybersecurity obligations for essential and important entities, including securing digital supply chains, of which AI services are part.
The Open-Source Ecosystem in 2026: Models Ready for Production
Mistral, LLaMA, Phi: The Leading Trio
The open-source LLM market has crossed a decisive threshold. According to Gartner, more than 60% of companies adopt an open-source LLM for at least one production use case, up from 25% in 2023. Three model families dominate private deployments.
Mistral is the European champion. Founded in Paris, Mistral AI reached $400 million in annual recurring revenue in January 2026, a 20-fold increase in twelve months. Mistral Large 2 scores 92% on HumanEval (code generation), outperforming most proprietary models. Its Mistral 7B and Mixtral models are distributed under Apache 2.0, allowing unlimited commercial use.
LLaMA (Meta) remains the world's most downloaded open-source model, with a 9% enterprise usage share according to market data. LLaMA 3.3 70B scores 86% on MMLU and 88.4% on HumanEval. Its community license allows commercial use up to 700 million monthly active users, a threshold reached only by the largest global platforms.
Phi (Microsoft) stands out for its performance-to-size ratio. Phi-4, with just 14 billion parameters, scores 80.4% on MATH, outperforming models five times larger. Its MIT license offers the broadest permissions in the market, without any usage restrictions.
Comparative Benchmarks: The Gap with Proprietary Models Is Narrowing
| Model | Parameters | MMLU | HumanEval | MATH | License |
|---|---|---|---|---|---|
| LLaMA 3.3 | 70B | 86.0% | 88.4% | 77.0% | Community (700M MAU) |
| Mistral Large 2 | — | 84.0% | 92.0% | — | Proprietary (API) |
| Mistral 7B | 7B | — | — | — | Apache 2.0 |
| Phi-4 | 14B | 84.8% | 82.6% | 80.4% | MIT |
| GPT-4o (reference) | — | ~88% | ~90% | ~76% | Proprietary (API) |
The finding is striking: LLaMA 3.3 70B rivals GPT-4o on most general-purpose benchmarks, while Mistral Large 2 surpasses it in code generation. For a company deploying an LLM on internal data—document analysis, legal assistance, report generation—these differences are negligible compared with the gain in sovereignty.
Specialized Models and Fine-Tuning
A decisive advantage of open-source models is the ability to fine-tune them on business data. A LLaMA 3.3 model fine-tuned on a French legal corpus will consistently outperform a general-purpose GPT-4o for drafting contract clauses or analyzing case law.
Efficient techniques such as LoRA and QLoRA allow a 70B model to be adapted with just 48 GB of VRAM, or a single A100 GPU. The process takes a few hours to a few days depending on training data volume.
Three Sovereign Deployment Architectures
Architecture 1: Fully On Premises
On-premises deployment runs the model on servers physically installed at your premises or in your data center. It offers maximum control.
Typical configuration for a 70B model:
- 2 NVIDIA A100 80 GB or H100 GPUs (140 GB VRAM in FP16)
- 256 GB system RAM
- Fast NVMe storage (1 TB minimum)
- Inference engine: vLLM or TGI (Hugging Face Text Generation Inference)
Advantages:
- No data leaves the company's physical perimeter
- Minimum latency (no external network transit)
- Full control of updates and configuration
- Hardware amortized over 3–5 years
Limitations:
- High initial investment (€50,000 or more for a production GPU server)
- Requires an internal team skilled in MLOps
- Scalability limited by available hardware
- Full responsibility for maintenance and physical security
Ideal profile: Large companies, regulated sectors (banking, healthcare, defense), organizations processing classified data.
Architecture 2: Dedicated VPC with a Sovereign Cloud Provider
A dedicated VPC (Virtual Private Cloud) offers a compromise between control and flexibility. Compute instances run on isolated infrastructure with contractual guarantees of nonshared resources and data location.
In France, three providers hold ANSSI's SecNumCloud qualification: OVHcloud, Outscale (Dassault Systèmes) and Scaleway. OVHcloud and Outscale already offer SecNumCloud-certified GPUs for LLM inference.
Typical configuration:
- Dedicated GPU instances (NVIDIA L40S, H100 or B200)
- Isolated private network (VPC) with encryption in transit and at rest
- No resources shared with other customers
- Hosting in French data centers, subject to European law
Advantages:
- Guaranteed legal and technical sovereignty
- Elasticity: scale up without buying hardware
- SecNumCloud certification for public procurement and sensitive sectors
- Predictable operating cost (usage-based or reserved billing)
Limitations:
- Significant monthly cost (€1,500–€3,000/month for a 70B model)
- Dependence on the cloud provider, even a sovereign one
- GPU service range sometimes narrower than hyperscalers'
Ideal profile: Mid-sized and large companies, public administrations and sectors subject to data residency requirements.
Architecture 3: Hybrid (On Premises + Sovereign Cloud)
A hybrid architecture combines on-premises deployment for the most sensitive data with overflow to a sovereign VPC to absorb load spikes.
Principle:
- The baseline model runs on premises for everyday usage
- A second identical model is instantiated in a sovereign VPC for spikes
- An intelligent router based on an API gateway directs requests according to sensitivity classification
- Data classified “confidential” remains exclusively on premises
- “Internal” data may be processed in the sovereign VPC
Ideal profile: Companies with variable request volumes and different sensitivity levels.
Comparison of the Three Architectures
| Criterion | On premises | Sovereign VPC | Hybrid |
|---|---|---|---|
| Data control | Maximum | High (contractual) | Maximum for sensitive data |
| Initial investment | €50,000+ | Low (OpEx) | €30,000+ |
| Recurring monthly cost | Electricity + maintenance | €1,500–€3,000/month | Variable |
| Scalability | Limited | High | High |
| Required skills | MLOps + infrastructure | Cloud + MLOps | Both |
| SecNumCloud certification | Not applicable | Yes | Partial |
| Latency | Minimal | Low (French network) | Variable |
| GDPR compliance | Native | Contractual | Mixed |
The Reference Technology Stack for Sovereign Deployment
Inference Engines: vLLM, the Production Standard
Inference engine choice determines deployment performance and reliability. In 2026, vLLM has established itself as the de facto standard for production environments.
Red Hat's 2025 benchmarks are decisive: vLLM reaches 793 tokens per second versus 41 for Ollama with equivalent configurations. This 19x difference comes from vLLM's architecture, designed from the outset for massive concurrency and memory optimization through PagedAttention.
| Engine | Throughput (tokens/s) | Use case | Production maturity |
|---|---|---|---|
| vLLM | 793 | Multiuser production | High |
| TGI (Hugging Face) | ~500 | Production, HF integration | High |
| Ollama | 41 | Local development, prototyping | Medium |
| llama.cpp | Variable | Edge, CPU-only, embedded | Good |
Recommendation: vLLM for production, Ollama for prototyping and developer testing.
Orchestration and Containerization
A production deployment extends beyond the inference engine. The complete stack includes:
- Kubernetes for GPU container orchestration and model lifecycle management
- NVIDIA Triton Inference Server or vLLM Production Stack for request routing, caching and observability
- Prometheus + Grafana for performance monitoring (latency, throughput, GPU utilization)
- MinIO or Ceph for object storage of models and fine-tuning data
- HashiCorp Vault for managing secrets and encryption keys
End-to-End Security
A sovereign LLM deployment's security rests on complementary layers.
Encryption: TLS 1.3 for communications, AES-256 encryption at rest for models and data. NVIDIA H100 and newer GPUs incorporate Confidential Computing, encrypting data even while processed in GPU memory.
Access control: Strong authentication (OIDC/SAML) combined with granular RBAC (Role-Based Access Control). Each team or application accesses only its assigned models and data.
Audit and traceability: Comprehensive logging of requests (without storing sensitive prompt contents), model access and administrative operations. Logs feed a SIEM for anomaly detection.
Prompt filtering: Application guardrails intercept requests before they reach the model, blocking prompt injection attempts and requests to extract training data.
Economic Analysis: When Self-Hosting Becomes Cost-Effective
Calculating Break-Even
A proprietary API's cost is measured in tokens consumed. A sovereign deployment combines initial investment, monthly operating costs and MLOps team payroll.
Concrete example: 30 million tokens/day
| Cost item | Proprietary API (GPT-4o) | Self-hosted LLaMA 70B (VPC) |
|---|---|---|
| Cost per million tokens | ~$3 (input) / ~$15 (output) | ~$0.10 (amortized) |
| Monthly cost (30M tokens/day) | ~$8,000–$12,000/month | $2,500/month (GPU + infrastructure) |
| Dedicated team | 0 (managed service) | 0.25 FTE MLOps engineer |
| Total monthly cost | ~$10,000 | ~$4,500 (infrastructure + personnel) |
| Break-even | — | Reached in 1–4 months |
According to an analysis published on arXiv (2025), GPU chips and personnel account for 70–80% of total self-hosted LLM deployment costs. Break-even varies with volume: from 50,000 daily requests, self-hosting consistently becomes more economical than APIs.
Hidden Costs to Anticipate
The basic calculation conceals several cost items decision-makers must include from project scoping.
Training and skills development: If the team lacks MLOps experience, allow 2–3 months of learning or recruitment of a specialist (€80K–€120K annually in Île-de-France).
Maintenance and updates: Open-source models evolve rapidly. Moving from LLaMA 3.1 to LLaMA 3.3 requires regression tests, revalidation on business data and potentially new fine-tuning.
Power and cooling: A server with 2 H100 GPUs consumes about 2–3 kW under load. Annually, that represents 15,000–25,000 kWh, or €3,000–€5,000 of electricity at French business rates.
Hardware obsolescence: The GPU innovation cycle is 18–24 months. An A100 investment today will be technically surpassed by B200s tomorrow, while remaining functional for inference for 4–5 years.
Implementation Guide: From Proof of Concept to Production
Phase 1: Prototyping (2–4 Weeks)
The objective is validating technical feasibility for one specific use case before heavy investment.
Key actions:
- Identify a high-value, low-risk use case (for example, automatic internal document summaries or knowledge base search)
- Install Ollama on a GPU-equipped developer workstation (RTX 4090: 28 GB VRAM, enough for Phi-4 14B)
- Test 2–3 models on a sample of real data, anonymized if necessary
- Compare response quality with a proprietary API on identical test cases
- Document results to decide whether to proceed to a pilot
Phase 2: Pilot (1–2 Months)
Deploy the selected model in an environment close to production with a restricted user group.
Key actions:
- Provision a dedicated GPU server (on premises or sovereign VPC)
- Deploy vLLM with production configuration (monitoring, logging, RBAC)
- Integrate the LLM into a business use case through an internal API (REST or gRPC)
- Train the pilot group (10–30 users) in model usage and limitations
- Collect qualitative feedback and quantitative metrics (latency, relevance, adoption rate)
Phase 3: Production Hardening (2–3 Months)
Moving to production means making the entire technical stack reliable.
Key actions:
- Containerize deployment through Kubernetes with automated scaling
- Establish end-to-end encryption and security auditing
- Implement application guardrails (prompt filtering, injection detection)
- Configure high availability (replication, automatic failover)
- Document operating procedures and train the Ops team
Phase 4: Continuous Optimization
Once in production, three optimization paths are available:
Fine-tuning on business data: Adapting the model to your internal corpus improves response relevance by 20–40% depending on use case, a gain impossible with a proprietary API that cannot be trained on your data.
RAG (Retrieval-Augmented Generation): Combining the LLM with a vector database (Qdrant, Weaviate, Milvus) fed by internal documents enables contextualized, referenced answers, reducing hallucinations by 40–60%.
Quantization: Compressing the model to FP8 or INT4 halves VRAM requirements without notable performance degradation. A LLaMA 70B quantized to INT4 runs on a single H100 GPU.
Concrete Use Cases: Four Business Scenarios
Scenario 1: Law Firm (50 Employees)
Need: Analyze contracts and case law without exposing client data to a third party.
Solution: Mistral 7B fine-tuned on French law, deployed on an on-premises server with an L40S GPU. Monthly cost: approximately €800 (amortization + electricity). The model powers a legal research assistant accessible through the firm's intranet.
Expected outcome: 60% less document research time, zero client data sent outside.
Scenario 2: Mid-Sized Industrial Company (2,000 Employees)
Need: Access technical documentation (maintenance manuals, product sheets, intervention histories) through an internal AI assistant.
Solution: LLaMA 3.3 70B deployed in a dedicated OVHcloud VPC (SecNumCloud), connected to a Qdrant vector database fed by 500,000 technical documents. RAG architecture provides referenced answers.
Expected outcome: Diagnosis time reduced by 45%, technicians autonomous on 80% of routine interventions.
Scenario 3: Healthtech Startup (30 People)
Need: Analyze medical reports to power a diagnostic support tool in strict compliance with health data regulations (HDS).
Solution: Phi-4 14B deployed with an HDS-certified health data hosting provider, with Confidential Computing encryption enabled. Monthly cost: approximately €600 on an L4 GPU.
Expected outcome: Process 10,000 reports/day with classification accuracy above 92%, native HDS compliance.
Scenario 4: Banking Group (20,000 Employees)
Need: Deploy a regulatory compliance assistant for risk and compliance teams without dependence on a US provider.
Solution: Hybrid architecture: Mistral Large 2 on premises for confidential documents, LLaMA 70B on an Outscale VPC (SecNumCloud) for routine requests. Automatic routing based on data classification.
Expected outcome: 70% of compliance requests processed in under 30 seconds, complete data processing auditability.
Mistakes to Avoid in Sovereign LLM Deployment
Undersizing Infrastructure
A 70B model requires 140 GB VRAM at FP16 precision. Running it on insufficient hardware leads to unacceptable response times (several minutes per request) or production crashes. Size for target load rather than current load.
Neglecting Fine-Tuning
A general-purpose model, however capable, does not know your business jargon, internal processes or industry constraints. Deploying it unchanged produces generic answers that disappoint users and kill adoption. Fine-tuning is a necessity for business use.
Ignoring Access Governance
Giving every employee uniform LLM access creates an internal leakage risk. An intern does not need to query ongoing M&A data. Implement RBAC from day one, with access levels aligned to data classification.
Forgetting Business Continuity
A failed GPU server means an unavailable AI service. Provide redundancy (automatic failover to a sovereign VPC if the on-premises server fails) and documented recovery procedures.
Predeployment Checklist
- Data passing through the LLM has been classified
- A DPIA (Data Protection Impact Assessment) has been completed if personal data is involved
- GPU sizing has been validated for the target load in 18 months
- An MLOps team has been identified (internal or partner)
- Model update procedures are documented
- Business continuity and disaster recovery plans are written
- Application guardrails are specified and tested
FAQ
What minimum budget is needed to deploy an enterprise LLM without public cloud? For a 14B model (Phi-4), allow €600 a month on an L4 GPU, sufficient for moderate volumes. For a production 70B model (LLaMA 3.3), budget €1,500–€3,000 a month in a sovereign VPC, or a €50,000 initial on-premises investment amortized over 3–5 years.
Are open-source models really as capable as GPT-4? On general-purpose benchmarks (MMLU, HumanEval), LLaMA 3.3 70B and Mistral Large 2 rival GPT-4o. The remaining gap mainly concerns multimodal tasks and complex long-chain reasoning. For most enterprise use cases—summarization, document search, report generation—the difference is negligible.
Do you need to recruit a specialist team to maintain an on-premises LLM? A production deployment requires at least 0.25 FTE of an MLOps engineer for routine maintenance. For organizations without internal skills, a specialized technical partner that sets up infrastructure and provides support is the most pragmatic approach.
Is SecNumCloud mandatory for a sovereign LLM deployment? No, except for certain public contracts and operators of vital importance (OIV). However, SecNumCloud qualification provides a recognized guarantee of legal and technical sovereignty and is becoming a differentiator in public-sector and large-enterprise tenders.
Can you fine-tune an open-source model on confidential data? Fine-tuning in a sovereign environment is precisely one of open-source models' major advantages. LoRA and QLoRA let you adapt a 70B model on a single A100 GPU (80 GB VRAM) in a few hours to a few days, without data leaving your infrastructure.
What is the security difference between on premises and a dedicated VPC? On premises, you control the entire chain: hardware, network and software. In a dedicated VPC, the cloud provider manages physical infrastructure but contractually guarantees isolation and data location. Effective security depends more on configuration quality than deployment model.
AI Coder Squad: Your Partner for Sovereign LLMs on Business Data
Building a secure LLM architecture that uses data without exposing it requires a rare combination: mastery of GPU inference infrastructure and the software engineering needed to integrate the model into business processes. These are exactly the projects we deliver every day.
AI Coder Squad designs custom applications and AI agents for companies that want to move fast without sacrificing quality, with senior developers and an AI-powered approach.
→ Start your project and discover how AI Coder Squad can accelerate your next delivery.