{"slug": "the-math-an-applied-ai-engineer-actually-uses-in-an-evaluation-loop", "title": "The Math an Applied AI Engineer Actually Uses in an Evaluation Loop", "summary": "An applied AI engineer outlines a practical evaluation loop for RAG and classification systems, arguing that aggregate accuracy metrics can mask increased product risk. The approach stores per-case metadata such as segment, expected source IDs, expected behavior, and failure cost, then computes retrieval metrics like recall@k and a weighted-risk score that assigns explicit costs to failure types including unsupported answers and unnecessary refusals.", "body_md": "A team compares two versions of an AI service. Average accuracy rises from 82% to 86%. The new version looks better, so releasing it seems like the obvious decision.\n\nThen an engineer separates the failures by type. The system now makes fewer mistakes on simple questions, but it is twice as likely to answer confidently without a source when the question concerns billing or contracts. The average improved while the product risk increased.\n\nThis is where an applied AI engineer needs mathematics. Not as an entrance exam, and not as an attempt to compress a university program into a few weeks. Math is useful here because it connects system behavior to an engineering decision: release the new version, return it for more work, or restrict one scenario.\n\nThe most practical way to begin is to build a small evaluation loop. Every formula in it should answer one engineering question.\n\nA weak test set often contains little more than a few pairs of `query -> good answer`. That is not enough for diagnosis.\n\nFor each evaluation case, store at least:\n\nA plain Python dictionary is enough to represent the first version:\n\n```\ncase = {\n    \"query\": \"Can I change the billing entity after payment?\",\n    \"segment\": \"billing\",\n    \"expected_source_ids\": [\"billing-policy-17\"],\n    \"expected_behavior\": \"answer_with_source\",\n    \"human_review_required\": False,\n    \"failure_cost\": 5,\n}\n```\n\nThe value of `failure_cost` is not a universal mathematical truth. It is an explicit team decision. A mistake in a low-risk reference answer might cost 1 point. An unsupported claim about a payment might cost 5. An action with irreversible consequences should be blocked by a separate policy instead of being reduced to another score.\n\nWithout this labeling, a metric quickly becomes a number with no owner.\n\nIn a RAG system, a bad answer can have several causes. The required document never entered the index. Retrieval did not return it near the top. The model received the correct source but ignored it. Or the source itself was obsolete.\n\nIf you measure only the final answer, those failures collapse into one result.\n\nA useful first retrieval metric is `recall@k`: the fraction of required sources that appears in the first `k` results.\n\n``` python\ndef recall_at_k(cases, retrieved_by_case, k):\n    recalls = []\n\n    for case in cases:\n        expected = set(case[\"expected_source_ids\"])\n        retrieved = set(retrieved_by_case[case[\"query\"]][:k])\n\n        if expected:\n            recalls.append(len(expected & retrieved) / len(expected))\n\n    if not recalls:\n        raise ValueError(\"No cases contain expected_source_ids\")\n\n    return sum(recalls) / len(recalls)\n```\n\nIf `recall@5` is low, changing the answer-generation prompt is premature. Check document splitting, metadata, filters, embeddings, and index freshness first.\n\nThis is where linear algebra becomes practical. An engineer needs to understand that an embedding is a vector and that cosine similarity, or another distance measure, is only a comparison rule. High similarity does not prove that a document answers the question. It only says that the selected representation and measure consider the objects similar.\n\nSuppose the system classifies incoming requests and routes them to a department. Accuracy alone is insufficient when classes are imbalanced or different mistakes have different consequences.\n\nAt minimum, distinguish between:\n\nYou can then calculate not only the fraction of correct results, but also weighted risk:\n\n```\nFAILURE_COST = {\n    \"false_positive\": 2,\n    \"missed_case\": 3,\n    \"unsupported_answer\": 5,\n    \"unnecessary_refusal\": 1,\n    \"wrong_escalation\": 2,\n}\n\ndef weighted_risk(results):\n    if not results:\n        raise ValueError(\"Results must not be empty\")\n\n    total_cost = sum(\n        FAILURE_COST[result[\"failure_type\"]]\n        for result in results\n        if result[\"failure_type\"] is not None\n    )\n\n    return total_cost / len(results)\n```\n\nThe process owner should agree on the cost of each failure. The formula does not make the business decision. It makes that decision visible and lets the team compare versions using the same rule.\n\nAn aggregate metric can improve simply because the test set contains many easy cases. Break the result down by a few meaningful segments, such as:\n\nEven a simple segment calculation tells you more than one overall number:\n\n``` python\nfrom collections import defaultdict\n\ndef accuracy_by_segment(results):\n    grouped = defaultdict(list)\n\n    for result in results:\n        grouped[result[\"segment\"]].append(result[\"is_correct\"])\n\n    return {\n        segment: sum(values) / len(values)\n        for segment, values in grouped.items()\n    }\n```\n\nIf the overall result improves while a sensitive segment gets worse, that regression should drive the discussion. An average should not override a safety boundary that the team agreed on in advance.\n\nAnother common mistake is to treat high confidence as evidence that an answer is correct.\n\nA better question is this: when the system reports confidence near 0.8, is it actually correct in roughly 80% of those cases?\n\nTo inspect this, group answers into confidence intervals and compare the average reported confidence with the observed fraction of correct results. A large gap indicates poor calibration.\n\nCalibration affects product rules directly:\n\nYou cannot choose the threshold separately from the cost of errors. In one product, an unnecessary refusal is almost harmless. In another, it destroys the product's main function.\n\nThirty evaluation cases are better than three demonstrations, but they may still represent real traffic poorly.\n\nThe minimum discipline is straightforward:\n\nAt this point, an engineer benefits from understanding samples, means, variance, and confidence intervals. The goal is not a more impressive report. It is to avoid calling random movement in a small sample an improvement.\n\nYou can build a useful first version without a complex platform:\n\nThis loop gives further mathematical study a concrete context. If retrieval fails, go deeper into vectors, distance measures, and ranking. If the model is confidently wrong, probability and calibration become the next step. If results vary substantially between runs, study sampling and variability more carefully.\n\nA useful sequence might be:\n\nThis is no longer an abstract curriculum. Each topic connects to a failure that the engineer can already observe in the evaluation loop.\n\nAn applied AI engineer does not need to wait for a complete mathematical education before measuring a system properly.\n\nStart by choosing the unit of evaluation, separating the stages of the system, naming the failure types, and connecting them to consequences. Mathematics then stops acting as a gate. It becomes the tool that explains why a new version is genuinely better, or why a better-looking average still does not justify a release.", "url": "https://wpnews.pro/news/the-math-an-applied-ai-engineer-actually-uses-in-an-evaluation-loop", "canonical_source": "https://dev.to/alexander_iwizard/the-math-an-applied-ai-engineer-actually-uses-in-an-evaluation-loop-4amg", "published_at": "2026-10-04 12:02:10+00:00", "updated_at": "2026-10-04 12:12:33.583451+00:00", "lang": "en", "topics": ["ai-research", "machine-learning", "large-language-models", "mlops", "ai-tools"], "entities": [], "also_reported_by": [], "takeaway": "Track recall@k and a weighted-risk score with explicit per-failure costs instead of average accuracy, since a version can improve accuracy while doubling unsupported answers on billing or contract questions.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/the-math-an-applied-ai-engineer-actually-uses-in-an-evaluation-loop", "markdown": "https://wpnews.pro/news/the-math-an-applied-ai-engineer-actually-uses-in-an-evaluation-loop.md", "text": "https://wpnews.pro/news/the-math-an-applied-ai-engineer-actually-uses-in-an-evaluation-loop.txt", "jsonld": "https://wpnews.pro/news/the-math-an-applied-ai-engineer-actually-uses-in-an-evaluation-loop.jsonld"}}