An AI application that works perfectly in a test environment can silently drift in production for weeks without anyone noticing. Deteriorating responses, soaring costs and hallucinations creeping into business workflows: symptoms often appear long after the damage is done. According to Gartner, 85% of AI projects fail, and the absence of production monitoring is among the structural causes.
Monitoring an AI application in production encompasses the processes and tools used to continuously collect, measure and analyze operating data from an artificial intelligence system. Unlike conventional application monitoring, it covers specific dimensions: the semantic quality of responses, unit cost per request, model drift and latency as perceived by the end user.
This article explains the four pillars of AI monitoring—latency, cost, quality and drift—and reviews tools for tracking them effectively.
TL;DR — Monitoring a production AI application requires tracking four dimensions: latency (p50, p95, p99), cost per request (tokens consumed × unit price), output quality (hallucinations, relevance, consistency) and drift (changes in input and output distributions). Platforms such as Datadog LLM Observability, LangSmith and Langfuse cover these needs at different levels of detail.
Why Conventional Monitoring Is Not Enough for an AI Application
The Limits of Traditional Observability
Standard application monitoring—HTTP response time, 5xx error rates, CPU and RAM consumption—covers the visible symptoms of failure. For an AI application, these metrics remain necessary but are fundamentally insufficient.
A language model can return a 200 status with a perfectly structured response containing invented information. The server is healthy from an infrastructure perspective, but the application produces a result harmful to the business. According to a consolidated DLF-NE analysis, LLM hallucinations caused estimated business losses exceeding USD 67 billion in 2024. The Air Canada case—in which the airline was held liable after its chatbot communicated a fictitious refund policy to a passenger—illustrates the legal and reputational consequences of this silent failure.
The Four Dimensions Specific to AI Monitoring
Monitoring a production AI application revolves around four pillars ignored by conventional monitoring:
| Dimension | What it measures | Why it is critical |
|---|---|---|
| Latency | Model response time: p50, p95, p99 | Direct impact on user experience and throughput |
| Cost | Tokens consumed, cost per request, cost per user | Undetected budget overruns can multiply the bill by 10 |
| Quality | Response relevance, accuracy and consistency | Invisible degradation can contaminate business decisions |
| Drift | Changes in input and output distributions | The model loses relevance as reality moves away from its training data |
Each dimension requires its own metrics, alert thresholds and response workflows. The following sections examine them individually.
Pillar 1: Latency—Measure What the User Really Experiences
Beyond Average Response Time
Average latency is misleading for AI applications. A model responding in 800 ms on average may still spike to 12 seconds on 5% of requests—precisely those involving complex prompts or long contexts. That is why experienced teams think in percentiles.
p50, the median, gives the typical response time. p95 captures degraded but common cases. p99 reveals extremes that cause users to abandon and applications to time out. For a B2B business application, p95 above 5 seconds signals an architecture or capacity problem.
Factors Affecting LLM Latency
Several variables affect production LLM latency, with relative importance varying by architecture:
Input context size. The longer the prompt—system instructions, conversation history and documents injected through RAG—the longer processing takes. A 500-token prompt processes in a fraction of the time required for 30,000 tokens.
Number of generated tokens. Generation is sequential: each token depends on the preceding one. A 2,000-token response inevitably takes longer than a 200-token response.
The model used. GPT-4o is faster than GPT-4 Turbo. Claude 3.5 Sonnet responds faster than Claude 3 Opus. Gemini Flash is designed for speed. Model selection balances latency, quality and cost.
Provider load. Cloud provider API latency varies with overall demand. Latency spikes of 3–5 times normal are common at peak times, especially for the most popular models.
Set Up Actionable Latency Monitoring
Effective latency monitoring for a production AI application must capture:
- Time to first token (TTFT): the delay before the first response token is generated, crucial for streaming interfaces.
- Time per output token (TPOT): sequential generation speed, determining the throughput perceived by users.
- End-to-end latency: total time including preprocessing—embedding and RAG retrieval—the model call and postprocessing.
Calibrate alert thresholds by request type. Simple classification and long-document generation have different performance expectations. Segmenting metrics by endpoint, task type and model avoids false positives.
Pillar 2: Cost per Request—The Metric Too Many Teams Discover Too Late
Understanding the Cost Structure of an LLM Call
The cost of an LLM API call has two components: input tokens and output tokens. In 2025–2026, prices vary considerably between models and providers:
| Model | Input cost per million tokens | Output cost per million tokens |
|---|---|---|
| GPT-4o | $2.50 | $10.00 |
| GPT-4o mini | $0.15 | $0.60 |
| Claude 3.5 Sonnet | $3.00 | $15.00 |
| Claude 3 Haiku | $0.25 | $1.25 |
| Gemini 2.0 Flash | $0.10 | $0.40 |
| Gemini Flash-Lite | $0.075 | $0.30 |
Sources: official provider prices, March 2026.
These price differences of up to 50-fold mean an unmonitored architecture choice can turn €500 per month into €25,000 for the same request volume. Prompt caching, available from most providers, reduces the cost of recurring input tokens by about 90%, but you still need to verify that it actually works in production.
The Most Common Cost Overruns
Three scenarios cause most exploding bills in production AI applications:
Silently expanding context. In a chatbot with conversation history, every exchange adds tokens to the context. Without truncation, the tenth interaction costs 10 times more than the first. In a RAG system, a poorly calibrated retriever injects too many chunks into the prompt—sometimes 15,000 tokens of context for a question requiring 2,000.
Autonomous agent loops. An AI agent repeatedly calling itself to refine a response can generate 20, 50 or even 100 API calls for a single user request. Without a per-session cost ceiling, one user can consume the monthly budget in minutes.
The wrong model for the task. Using GPT-4o to classify support tickets when GPT-4o mini would deliver the same result at 6% of the cost. This overprovisioning goes unnoticed without granular per-endpoint monitoring.
Build a Cost-Tracking Dashboard
Effective cost monitoring aggregates data at several levels:
- Per request: input tokens, output tokens, unit cost, model used and cache hit/miss.
- Per user or session: cumulative cost, request count and average cost per interaction.
- Per feature: does the RAG module cost more than summarization? Does the internal chatbot consume more than the customer assistant?
- Per period: daily, weekly and monthly trends, with anomaly detection.
Configure cost alerts at two levels: a warning threshold, such as reaching 80% of the monthly budget, and a critical threshold, such as exceeding the budget or a per-request cost spike greater than 3 times the median.
Pillar 3: Output Quality—AI Monitoring's Most Complex Challenge
Why Quality Is Difficult to Measure Automatically
Unlike latency or cost, AI response quality is not an objective number. A response can be grammatically correct and well structured yet factually wrong. It can be accurate but inappropriate to the context. It can satisfy a technical user while confusing a business user.
According to a Stanford HAI study, LLMs hallucinate 69–88% of the time on legal queries. Even on less specialized tasks, measured hallucination rates range from 0.7% for Gemini 2.0 Flash to nearly 30% for small open-source models. These figures demonstrate that no production AI application can do without continuous quality monitoring.
Quality Metrics to Instrument
Monitoring output quality in a production AI application relies on several metric families:
Hallucination detection metrics. Tools such as Vellum, Langfuse and LLM-as-judge evaluators compare generated responses with sources supplied in context. Groundedness—the proportion of the response supported by source documents—is a key metric for RAG systems.
Relevance metrics. Does the response answer the question? Semantic similarity scores between question and answer, or automatic evaluations by a judge model, help detect off-topic responses.
Compliance metrics. Does the response respect business constraints? Expected format—JSON, table or structured text—length, tone, and absence of sensitive or prohibited content. These checks often use deterministic rules rather than models.
User satisfaction metrics. Implicit signals—rephrasing rate, abandonment rate and time spent on the response—and explicit signals—thumbs up/down and written feedback—complement automated evaluations.
Build a Continuous Evaluation Pipeline
A production quality evaluation pipeline operates in three layers:
Layer 1 — Synchronous checks. Before returning the response: format checks, toxic-content detection and validation of business constraints. These add a few milliseconds of latency but block clearly defective responses.
Layer 2 — Asynchronous evaluations. After sending the response: LLM-as-judge scoring, groundedness calculations and hallucination detection. These feed dashboards and alerts without affecting latency.
Layer 3 — Sampled human review. A percentage of responses—2–10%, depending on criticality—is routed to human review queues. According to DextraLabs, 76% of companies deploying LLMs in production now include human-in-the-loop review. Human annotations feed evaluation datasets and help recalibrate automated evaluators.
Practical guide: Five Quality Warning Signs to Watch
- The thumbs-down rate exceeds 15% over 24 hours
- The rephrasing rate—users asking the same question differently—increases by more than 20%
- Average groundedness falls below 0.7 for RAG responses
- The model generates responses significantly longer or shorter than the historical median
- The rate of responses containing “I don't know” or equivalent exceeds the normal threshold
Pillar 4: Drift—When the Model Loses Touch with Reality
Understanding Types of Drift
Drift is the gradual divergence between the conditions in which a model was designed or calibrated and actual production conditions. Three categories affect AI applications:
Data drift: changes in input data. Users change how they phrase queries. Topic distributions evolve. New terms and concepts appear. An HR assistant calibrated for paid-leave questions suddenly receives a wave of queries about post-COVID remote work: query embeddings move away from the historical centroid.
Concept drift. The relationship between inputs and correct outputs changes. A regulation is amended, an internal policy evolves or a product is updated. The model continues answering from the old reference framework. Consequences are particularly severe in legal, tax and regulatory contexts.
Label drift. The distribution of expected outcomes changes. A support ticket classifier trained on 60% technical and 40% billing tickets finds those proportions reversed. Its decision thresholds are no longer calibrated.
Detect Drift in Production
Drift detection relies on statistical comparison between baseline distributions and those observed in production. Different metrics suit different data types:
| Feature type | Recommended metrics |
|---|---|
| Numerical | PSI—Population Stability Index, Kolmogorov–Smirnov test, Wasserstein distance |
| Categorical | Chi-squared test, tracking top-k categories |
| Text / embeddings | Centroid shift, average cosine distance |
| Model outputs | Prediction distribution, confidence drift |
Observation uses three time horizons: a short window of 1 hour to 1 day for abrupt changes, a medium window of 7 days to smooth noise, and a long window of 30 days for gradual drift.
A Drift Response Strategy
Detection is not enough; you must know how to respond. A two-level alert system is recommended:
Warning level. Drift is detected but remains within acceptable bounds. Action: investigate, document and increase monitoring. Does drift persist over N consecutive windows, or is it a one-off artifact?
Critical level. Drift exceeds tolerance thresholds and affects quality metrics. Action: switch to a fallback model, activate systematic human-in-the-loop review, and trigger retraining or prompt recalibration.
The most reliable approach combines scheduled monthly or quarterly retraining or recalibration with retraining triggered by drift or performance-degradation alerts. For applications using LLM APIs without fine-tuning, drift response mainly involves updating system prompts, RAG knowledge bases and guardrails.
Production AI Monitoring Tools: A 2026 Overview
Specialized LLM Platforms
The AI observability market is growing rapidly: according to Precedence Research, the global MLOps market is valued at USD 4.38 billion in 2026, with a 37% compound annual growth rate. Several platforms focus on LLM application monitoring:
LangSmith (LangChain). A unified observability, evaluation and prompt engineering platform. Its strength is Annotation Queues, enabling business experts to review, annotate and correct production traces. Annotations feed evaluation datasets directly. LangSmith supports offline evaluations of known scenarios before deployment, online evaluations of real-time production data, and multi-turn conversation evaluations. It is framework-agnostic.
Langfuse. An open-source LLM observability solution providing detailed prompt-chain tracing, cost and token tracking, and quality scoring. Its main advantage is self-hosting, addressing data sovereignty requirements—a critical issue for European companies subject to GDPR.
Arize Phoenix. An ML and LLM observability platform with advanced embedding drift detection, query cluster visualization and identification of areas where performance deteriorates.
Traditional Observability Providers Are Adapting
Datadog LLM Observability. Datadog launched dedicated LLM monitoring integrated natively with its infrastructure and application observability ecosystem. Its Python SDK automatically traces LLM calls—OpenAI, Anthropic, AWS Bedrock and LangChain—and captures latency, errors and token consumption. AI Agent Monitoring, presented at DASH 2025, maps AI agent decision paths—inputs, tool calls, agent interactions and outputs—in an interactive graph. Pricing: $8 per month per 10,000 monitored LLM requests, billed annually, with a minimum of 100,000 requests per month, creating an effective $80 monthly minimum.
New Relic and Dynatrace. Both platforms have also developed AI observability modules, building on existing enterprise stack integrations. Their advantage is a unified infrastructure, application and AI view in a single dashboard.
Criteria for Choosing an AI Monitoring Platform
| Criterion | Specialized platform: LangSmith, Langfuse | General platform + AI module: Datadog, New Relic |
|---|---|---|
| LLM tracing depth | ★★★★★ | ★★★☆☆ |
| Infrastructure integration | ★★☆☆☆ | ★★★★★ |
| Integrated quality evaluation | ★★★★★ | ★★★☆☆ |
| Self-hosting available | ★★★★☆ (Langfuse) | ★☆☆☆☆ |
| Learning curve | Moderate | Low for existing customers |
| Entry cost | Freemium / open source | $80/month minimum |
The choice depends on your maturity. If you are starting with a first LLM project, a specialized platform offers greater depth. If you already manage applications monitored by Datadog or New Relic, adding the LLM module reduces tool fragmentation.
Building an AI Monitoring Strategy: A Step-by-Step Approach
Step 1 — Instrument from Day One
AI monitoring is not something to bolt onto a production application afterward. Design it during development. Trace every model call with a unique trace ID linking the user request, submitted prompt, received response, consumed tokens, latency and cost.
Observability SDKs such as Langfuse, LangSmith and Datadog integrate in a few lines of code. The effort is minimal during construction and considerable when catching up later.
Step 2 — Define AI-Specific SLOs
Service Level Objectives for an AI application go beyond uptime. They should cover all four pillars:
- Latency SLO: “95% of classification requests are answered in under 2 seconds”
- Cost SLO: “Average cost per conversation does not exceed €0.15”
- Quality SLO: “Groundedness remains above 0.8 for RAG responses”
- Drift SLO: “No critical drift alert remains unaddressed for more than 48 hours”
These SLOs form the quality agreement between the technical team and business stakeholders. They turn monitoring from a technical exercise into a governance tool.
Step 3 — Automate Incident Responses
Monitoring that produces alerts without response workflows is noise. Associate every alert with a runbook:
p95 latency above threshold. Check provider load, context size and cache availability. If the provider is responsible, activate routing to a fallback model.
Cost overrun. Identify the responsible endpoint or user. Check for expanding context through per-request token logs. Activate rate limiting if necessary.
Quality degradation. Increase human sampling. Analyze affected requests. Adjust prompts or retrieval settings if the issue is systemic.
Drift alert. Compare current distributions with the baseline. If drift is confirmed across several windows, schedule updates to prompts, the RAG knowledge base or the model.
Step 4 — Report to Stakeholders
AI monitoring must produce reports that nontechnical people can read. A CIO or business director does not read a p99 chart: they want to know whether the application delivers on its promises, what it costs and whether users are satisfied.
A typical weekly report covers request volume, total cost and cost by feature, user satisfaction, detected incidents and their resolution, and changes in quality metrics. This report turns technical monitoring into a source of organizational confidence.
The Most Common AI Monitoring Mistakes
Monitoring Infrastructure Only
This is the most widespread mistake. Operations teams watch CPU, memory and network activity and consider the application healthy while servers are running. But a hallucinating LLM returns a 200 status. Infrastructure monitoring is necessary and never sufficient.
Ignoring Cost Until the Bill Arrives
According to PricePerToken, LLM API prices range from $0.075 to $75 per million tokens depending on the model. Without granular monitoring, a poorly optimized prompt or looping agent can multiply the monthly bill by 10–50. Tools such as LiteLLM or Bifrost track costs in real time and set budgets by project, team or endpoint.
Evaluating Quality Only at Launch
AI application quality naturally degrades over time: real-world data evolves, users change their habits and providers update models. A one-time launch evaluation gives a false sense of security. Continuous evaluation, both automated and human, is the only viable production approach.
Alerting Without Context
Receiving 200 alerts a day without prioritization or context is worse than receiving none. Alert fatigue leads teams to ignore critical signals. Enrich each alert with the endpoint, user, model and trend. Modern observability platforms correlate alerts across pillars: increasing latency alongside declining quality points to a provider issue rather than an application issue.
FAQ
What Is the Difference Between Monitoring and Observability for an AI Application?
Monitoring collects and displays predefined metrics such as latency, cost and error rate. Observability goes further: it lets you explore individual traces and understand why an anomaly occurred by following a request's complete path—from prompt to result, through tool calls and retrieval steps.
What Budget Should You Allow for Monitoring a Production AI Application?
Cost varies with volume and tool choice. Langfuse is open source and self-hostable. LangSmith offers a free tier of up to 5,000 traces per month. Datadog LLM Observability starts at $80/month for 100,000 requests. For an SME with moderate usage of 50,000–200,000 requests per month, allow between €0 for open source and $300/month.
How Often Should You Check an AI Model for Drift?
Drift detection should be automated and continuous. Recommended practice uses three observation windows: short, 1 hour–1 day, for abrupt changes; medium, 7 days, to smooth noise; and long, 30 days, for gradual drift. Alerts should trigger only when drift persists over several consecutive windows.
Does Monitoring Differ Between Fine-Tuned Models and API-Based AI Applications?
Yes. An application using a fine-tuned model requires additional monitoring of the model itself, including training metrics and post-deployment degradation. An application consuming LLMs through APIs focuses more on prompts, cost per token and response quality, with the provider managing the model.
What Warning Signs Indicate Insufficient AI Monitoring?
Five signs should raise concern: API bills rising without a change in request volume; user complaints about quality without data to investigate; inability to answer “what does each feature cost?”; surprises when providers update models; and no baseline for comparing current with historical performance.
AI Coder Squad: Monitoring Built into Your AI Applications from the Start
Designing an AI application without a monitoring strategy means operating blind. At AI Coder Squad, observability is integrated from the first lines of code—LLM call tracing, cost tracking, quality evaluation and drift detection—so every delivered application can be governed in production.
AI Coder Squad designs custom applications and AI agents for businesses that want to move quickly without sacrificing quality—with senior developers and an AI-powered approach.
→ Start your project and discover how AI Coder Squad can accelerate your next delivery.