The Math an Applied AI Engineer Actually Uses in an Evaluation Loop An applied AI engineer outlines a practical evaluation loop for RAG and classification systems, arguing that aggregate accuracy metrics can mask increased product risk. The approach stores per-case metadata such as segment, expected source IDs, expected behavior, and failure cost, then computes retrieval metrics like recall@k and a weighted-risk score that assigns explicit costs to failure types including unsupported answers and unnecessary refusals. A team compares two versions of an AI service. Average accuracy rises from 82% to 86%. The new version looks better, so releasing it seems like the obvious decision. Then an engineer separates the failures by type. The system now makes fewer mistakes on simple questions, but it is twice as likely to answer confidently without a source when the question concerns billing or contracts. The average improved while the product risk increased. This is where an applied AI engineer needs mathematics. Not as an entrance exam, and not as an attempt to compress a university program into a few weeks. Math is useful here because it connects system behavior to an engineering decision: release the new version, return it for more work, or restrict one scenario. The most practical way to begin is to build a small evaluation loop. Every formula in it should answer one engineering question. A weak test set often contains little more than a few pairs of query - good answer . That is not enough for diagnosis. For each evaluation case, store at least: A plain Python dictionary is enough to represent the first version: case = { "query": "Can I change the billing entity after payment?", "segment": "billing", "expected source ids": "billing-policy-17" , "expected behavior": "answer with source", "human review required": False, "failure cost": 5, } The value of failure cost is not a universal mathematical truth. It is an explicit team decision. A mistake in a low-risk reference answer might cost 1 point. An unsupported claim about a payment might cost 5. An action with irreversible consequences should be blocked by a separate policy instead of being reduced to another score. Without this labeling, a metric quickly becomes a number with no owner. In a RAG system, a bad answer can have several causes. The required document never entered the index. Retrieval did not return it near the top. The model received the correct source but ignored it. Or the source itself was obsolete. If you measure only the final answer, those failures collapse into one result. A useful first retrieval metric is recall@k : the fraction of required sources that appears in the first k results. python def recall at k cases, retrieved by case, k : recalls = for case in cases: expected = set case "expected source ids" retrieved = set retrieved by case case "query" :k if expected: recalls.append len expected & retrieved / len expected if not recalls: raise ValueError "No cases contain expected source ids" return sum recalls / len recalls If recall@5 is low, changing the answer-generation prompt is premature. Check document splitting, metadata, filters, embeddings, and index freshness first. This is where linear algebra becomes practical. An engineer needs to understand that an embedding is a vector and that cosine similarity, or another distance measure, is only a comparison rule. High similarity does not prove that a document answers the question. It only says that the selected representation and measure consider the objects similar. Suppose the system classifies incoming requests and routes them to a department. Accuracy alone is insufficient when classes are imbalanced or different mistakes have different consequences. At minimum, distinguish between: You can then calculate not only the fraction of correct results, but also weighted risk: FAILURE COST = { "false positive": 2, "missed case": 3, "unsupported answer": 5, "unnecessary refusal": 1, "wrong escalation": 2, } def weighted risk results : if not results: raise ValueError "Results must not be empty" total cost = sum FAILURE COST result "failure type" for result in results if result "failure type" is not None return total cost / len results The process owner should agree on the cost of each failure. The formula does not make the business decision. It makes that decision visible and lets the team compare versions using the same rule. An aggregate metric can improve simply because the test set contains many easy cases. Break the result down by a few meaningful segments, such as: Even a simple segment calculation tells you more than one overall number: python from collections import defaultdict def accuracy by segment results : grouped = defaultdict list for result in results: grouped result "segment" .append result "is correct" return { segment: sum values / len values for segment, values in grouped.items } If the overall result improves while a sensitive segment gets worse, that regression should drive the discussion. An average should not override a safety boundary that the team agreed on in advance. Another common mistake is to treat high confidence as evidence that an answer is correct. A better question is this: when the system reports confidence near 0.8, is it actually correct in roughly 80% of those cases? To inspect this, group answers into confidence intervals and compare the average reported confidence with the observed fraction of correct results. A large gap indicates poor calibration. Calibration affects product rules directly: You cannot choose the threshold separately from the cost of errors. In one product, an unnecessary refusal is almost harmless. In another, it destroys the product's main function. Thirty evaluation cases are better than three demonstrations, but they may still represent real traffic poorly. The minimum discipline is straightforward: At this point, an engineer benefits from understanding samples, means, variance, and confidence intervals. The goal is not a more impressive report. It is to avoid calling random movement in a small sample an improvement. You can build a useful first version without a complex platform: This loop gives further mathematical study a concrete context. If retrieval fails, go deeper into vectors, distance measures, and ranking. If the model is confidently wrong, probability and calibration become the next step. If results vary substantially between runs, study sampling and variability more carefully. A useful sequence might be: This is no longer an abstract curriculum. Each topic connects to a failure that the engineer can already observe in the evaluation loop. An applied AI engineer does not need to wait for a complete mathematical education before measuring a system properly. Start by choosing the unit of evaluation, separating the stages of the system, naming the failure types, and connecting them to consequences. Mathematics then stops acting as a gate. It becomes the tool that explains why a new version is genuinely better, or why a better-looking average still does not justify a release.