# #Your Model Is Not the System: Building an MLOps Feedback Loop for Production AI > Source: > Published: 2026-11-06 07:10:28+00:00 A machine learning model can achieve excellent evaluation results and still fail in production. The model hasn't changed. The code hasn't changed. The deployment is healthy. But prediction quality is declining. Maybe the input distribution has shifted. Maybe an upstream data pipeline changed its schema. Maybe the business environment has evolved. Or, in a generative AI application, the model provider updated its API, a prompt changed, or a retrieval pipeline started returning different context. This is why deploying a model is not the same as operating an AI system. At **SotaTek**, we approach MLOps as a continuous engineering discipline: version what affects behavior, observe production signals, identify the source of degradation, and decide whether to fix the data, update the model, roll back, or redesign the system. Let's look at what that actually requires. Model monitoring is one component of MLOps, not the entire operating model. A production MLOps system needs to connect experimentation, data pipelines, model training, evaluation, deployment, monitoring, and incident response. | Capability | Engineering responsibility | |---|---| | Experiment tracking | Record datasets, parameters, metrics, and results | | Model registry | Manage approved model artifacts and versions | | CI/CD | Test and promote code and model changes | | Continuous training | Rebuild models when evidence justifies it | | Data pipelines | Maintain reliable training and inference inputs | | Evaluation | Validate candidate models against release criteria | | Monitoring | Detect abnormal production behavior | | Observability | Investigate why behavior changed | | Rollback | Restore a previously approved version | A dashboard might tell you that accuracy has dropped. An operational MLOps system should help answer: The objective is not just to detect a problem. It's to make the next engineering action clear. For conventional machine learning, teams often focus on the model artifact, features, training dataset, and inference environment. Generative AI expands the operational surface. A production LLM application may depend on: Consider an enterprise support assistant. Its behavior might change even if the application code remains untouched: ``` Release 2.4.0 | +-- LLM: model-v1 +-- Prompt: support-v18 +-- Embedding: embedding-v3 +-- Reranker: reranker-v1 +-- Retrieval top_k: 5 +-- Guardrails: policy-v4 Release 2.4.1 | +-- LLM: model-v2 +-- Prompt: support-v18 +-- Embedding: embedding-v3 +-- Reranker: reranker-v2 +-- Retrieval top_k: 8 +-- Guardrails: policy-v4 ``` Both releases may use the same application code, but they do not represent the same AI system. If answer quality changes between these releases, the team needs enough metadata to identify which component contributed to the regression. This is why modern AI operations must version more than model checkpoints. A reproducible release should capture the complete configuration required to understand and reproduce its behavior. For conventional ML, track: For LLM applications, also track: A release manifest could look like this: ``` release: support-agent-2.4.1 model: provider: approved-provider version: model-2026-09 prompt: version: support-system-v18 retrieval: embedding: embedding-v4 reranker: reranker-v2 top_k: 8 evaluation: groundedness: 0.976 task_success: 0.943 critical_failures: 0 deployment: previous_version: support-agent-2.4.0 rollback_enabled: true ``` These values are illustrative configuration, not measured SotaTek performance data. The important part is maintaining lineage between the release, its dependencies, and its evaluation results. Without that lineage, regression analysis becomes guesswork. A production ML lifecycle should make every transition explicit. ``` Experiment | v Code + Data + Parameters | v Automated Training Pipeline | +-- Data Validation +-- Training +-- Evaluation | v Model Registry | v Release Gate | v Staging | +-- Integration Tests +-- Performance Tests +-- Acceptance Tests | v Production | v Telemetry + Monitoring | v Alert / Evaluation | +-------------------+ | | v v Rollback Retrain / Improve ``` CI validates code and configuration changes. CD promotes approved artifacts through deployment environments. Continuous training (CT) can rebuild models when production evidence indicates that the existing model is no longer suitable. But automatic retraining should not be the default response to every drift alert. A distribution change is evidence that something changed. It does not prove that retraining will solve the problem. One of the most important distinctions in production monitoring is understanding what actually changed. There are several different failure signals. **Data quality issues** Inputs may contain missing values, invalid categories, unexpected timestamps, schema errors, or out-of-range values. The data pipeline may be broken even when the underlying business distribution is stable. **Training-serving skew** The training and inference pipelines produce systematically different features. For example, a feature may be calculated using one normalization rule during training and another during production. **Data drift** The statistical distribution of production inputs changes compared with the chosen baseline. A credit model trained primarily on salaried employees may begin receiving a larger proportion of self-employed applicants. **Prediction drift** The distribution of model outputs changes. A fraud model may suddenly classify more transactions as suspicious. That could indicate degradation, but it could also mean fraud activity has genuinely increased. **Concept drift** The relationship between inputs and outcomes changes. Consumer purchasing patterns that previously predicted churn may stop working after a competitor changes its pricing strategy. Feature distributions can remain relatively stable while model performance deteriorates. These signals should not trigger the same automated response. A useful production response flow is: ``` Drift Detected | v Validate Data Quality | v Measure Actual Model Impact | v Root-Cause Analysis | +-- Data error --------> Fix Pipeline | +-- Temporary shift --> Continue Monitoring | +-- Model decay -------> Evaluate Retraining | +-- Structural shift --> Redesign Model ``` The key is to distinguish a change in input distribution from a change in actual model quality. When ground-truth labels become available, compare predictions against real outcomes. For a classification model, relevant metrics may include: For regression, consider MAE or RMSE. The evaluation metric should reflect the actual cost of model errors. A fraud detection system, for example, may care more about missed fraud than overall accuracy. A drift alert is a reason to investigate, not an automatic instruction to retrain. Traditional ML monitoring focuses on data quality, prediction distributions, model performance, and infrastructure. LLM applications need additional signals. | Layer | Production signals | |---|---| | Model | Answer correctness, refusal rate | | Retrieval | Recall\@K, relevance, reranker quality | | Generation | Groundedness, hallucination, citation accuracy | | Performance | Time to first token, end-to-end latency | | Economics | Input/output tokens, cost per request | | Safety | Guardrail violations, sensitive-data exposure | | Workflow | Human escalation, task completion | For RAG systems, retrieval and generation should be monitored separately. If groundedness falls, there are at least two possible explanations: Those failures require different interventions. A useful trace structure might look like: ``` { "request_id": "req-8f32", "model_version": "model-v2", "prompt_version": "support-v18", "retrieval": { "embedding_version": "embedding-v4", "top_k": 8, "documents_retrieved": 8, "latency_ms": 145 }, "generation": { "input_tokens": 3200, "output_tokens": 420, "latency_ms": 1800 }, "evaluation": { "groundedness": 0.94, "citation_valid": true } } ``` This is an illustrative telemetry schema. A production implementation should also apply data minimization, access controls, retention policies, and redaction for sensitive inputs and outputs. The purpose of tracing is to reconstruct the execution path without exposing more user data than necessary. For an AI agent, the final answer is only part of the execution. An agent may retrieve information, select tools, invoke APIs, retry failed operations, request approval, and execute business actions. Monitoring only the final response misses most of the operational behavior. Consider this workflow: ``` User Request | v Agent | +-- Retrieval | +-- Tool A | +-- Tool B | +-- Reason / Decide | v Business Action | v Final Response ``` Useful production signals include: An agent that reaches the correct result after 25 unnecessary tool calls may be technically successful but operationally inefficient. For many enterprise workflows, cost per successful task is more meaningful than cost per token. More importantly, an agent that executes an unauthorized action must trigger a security response, not just a performance alert. The strongest monitoring systems do not stop at generating alerts. They connect production signals to a defined response. ``` Production AI | v Telemetry Layer | v Monitoring / Evaluation | v Threshold / Anomaly | v Alerting | v Incident Triage | +-- Observe | +-- Reduce Traffic | +-- Rollback | +-- Retrain | +-- Human Escalation | v Re-evaluation | v Controlled Release ``` A threshold has little meaning without a useful comparison point. Depending on the metric, baselines may include: A retailer comparing December traffic with October may see significant distribution changes that are simply seasonal behavior. Monitoring design requires domain knowledge. Monitoring everything is not necessarily good monitoring. Prioritize signals that correspond to meaningful business, reliability, or safety risks. | Alert | Severity | Response | |---|---|---| | High GPU utilization | Low | Capacity review | | Data drift above threshold | Medium | Investigate | | Model accuracy below SLA | High | Block next rollout | | Sensitive-data exposure | Critical | Disable affected workflow | | Unauthorized agent action | Critical | Stop agent and initiate incident response | Monitoring tells you that something is wrong. Observability helps you understand why. A new model should rarely receive 100% of production traffic immediately when failures can materially affect customers or operations. Common deployment strategies include: A canary release exposes a small percentage of production traffic to a candidate model while comparing its quality, latency, errors, and business metrics with the incumbent. Production failures should improve the next release. A useful feedback loop is: ``` Production Failure | v Root Cause | v New Evaluation Case | v Fix | v Regression Test | v Release ``` For example, if an LLM assistant repeatedly cites outdated policy documents, add those scenarios to the evaluation suite. If an agent attempts an unauthorized tool call, add a test that verifies the permission boundary. Over time, the evaluation dataset becomes a record of what production has taught the engineering team. This is why model evaluation and MLOps should be designed together rather than operated by disconnected teams. An AI system can be technically healthy while failing commercially. Imagine a support assistant with: But employees ignore it because it adds two minutes to every workflow. From an infrastructure perspective, the system looks healthy. From a business perspective, it is failing. Connect AI telemetry to metrics such as: The goal is not to maximize model accuracy in isolation. It is to maintain a reliable system that delivers the expected business outcome. Before calling an AI system production-ready, verify: **Versioning and reproducibility** **Evaluation and release** **Monitoring and observability** **Generative AI and agents** **Incident response** If several of these capabilities are missing, the organization may have deployed a model, but it does not yet have a mature MLOps capability. MLOps is the operating discipline that separates an AI demo from an AI system an enterprise can depend on. Models encounter new data. User behavior changes. Infrastructure fails. Model providers update. Prompts evolve. Knowledge bases become stale. Agents gain access to more tools. The right response is a continuous lifecycle: **Build → Evaluate → Deploy → Monitor → Diagnose → Improve → Re-evaluate** At SotaTek, we treat MLOps, LLMOps, and AI operations as part of the production architecture, connecting model evaluation, versioning, monitoring, integration, and operational controls rather than adding monitoring as an afterthought. The best MLOps teams are not necessarily the teams that retrain models most often. They are the teams that know when behavior changed, why it changed, whether it matters, and what action should follow. For a deeper look at the monitoring framework, production lifecycle, and enterprise implementation approach, read our [AI MLOps and Monitoring guide](https://www.sotatek.com/blogs/ai-and-machine-learning/ai-mlops-and-monitoring/).