Pick the LLM eval metric before you build the dashboard

A developer argues that LLM eval metrics should be chosen to match the feature's shape before any dashboard is built: F1 (or split precision and recall) for imbalanced classifiers, AUC-ROC for threshold-free model comparison, perplexity only as a like-for-like development signal for autoregressive models, and faithfulness paired with context and answer relevance for RAG systems. The writeup stresses that perplexity does not predict task success and that faithfulness can hold even when the underlying sources are wrong.

You can ship a polished eval dashboard and still learn nothing about whether the product works. A common way to get there is picking a number because it looked authoritative somewhere else, then discovering that production behavior and the chart tell different stories. Before you wire graphs, name the feature shape classifier, ranker, generator, or grounded answerer and choose a metric that matches that shape. If your LLM feature ends in a yes or no my examples: toxic or not, escalate or not, intent A vs B , you have a classification problem even when the model reasons in natural language. For imbalanced labels, accuracy misleads: a model that always predicts the majority class can score high while catching none of the cases you care about. The F1 score https://chainofthought.show/glossary/f1-score/ is the harmonic mean of precision and recall, so it stays low if either side is weak. A system cannot game F1 by maximizing precision while missing every positive, or by calling everything positive to inflate recall. I still split precision and recall when I need to know which failure mode hurts users. F1 weights the two equally; your error costs might not. When the model outputs a score and you have not fixed the decision threshold, AUC-ROC https://chainofthought.show/glossary/auc-roc/ is a reasonable summary for comparing models on the same data. It reflects how well positives rank above negatives as you sweep thresholds. AUC near 0.5 is random separation on that set; near 1.0 is strong separation. On rare positives fraud, abuse, defects , AUC can look strong while most positive predictions are still wrong, because a small false positive rate on a huge negative pool still produces many false alarms. When positives are rare, report precision and recall, or F1 at the threshold you plan to ship, alongside AUC. Autoregressive models assign probabilities to next tokens. Perplexity https://chainofthought.show/glossary/perplexity/ rolls that into one number: lower means the held-out text surprised the model less. It helps during training and when you compare models on the same unseen text with the same tokenizer. Change the tokenizer or let test data leak into training, and the comparison breaks. Perplexity is not well defined for masked language models the way it is for standard autoregressive LMs, and it measures prediction on text, not success on your task. Better perplexity does not guarantee better outcomes on the work you ship. A model can assign high probability to fluent continuations and still hallucinate, ignore instructions, or fail your rubric. I use perplexity as a development signal for like-for-like model comparison, not as proof a chat, copilot, or agent is ready for users. Generators need task evals: rubrics, human review, judges where that is appropriate, and checks tied to the user request. Only measuring the application end to end tells you whether the feature works. RAG and citation-backed answers raise a different question: did the model stay inside the retrieved context, or did it invent details? Faithfulness https://chainofthought.show/glossary/faithfulness/ sometimes called groundedness checks whether claims in the answer are supported by the supplied sources. It targets hallucination in the sense of adding material that is not in the context. Faithfulness is separate from whether the answer is correct. An answer can be fully faithful to the wrong source documents. Pair it with context relevance was the right material retrieved? and answer relevance did the model address the question? , not as a single headline KPI. If you want your model to do something different, you don't go and perform brain surgery on the neurons. You don't reinvent the architecture of the algorithm. Let me go and invent a new post-transformer architecture. No, you get data. Alex Ratner, Snorkel AI, on Chain of Thought ep 57 Ratner's point is that data is the primary tool for changing a model, even if everything matters. Measurement data works the same way: pick the instrument to match the feature, then invest in labels and rubrics that expose gaps before users do. For binary classifier features, I start with a labelled holdout set that reflects production skew. Choose a threshold on a validation slice using the false positive and false negative costs you accept, then report F1 at that threshold for the story stakeholders will feel. Keep AUC on scored examples to compare models before you lock a cutoff. Minimal Python sketch not production code : Sketch: binary labels, thresholded preds, and scores on a holdout set. from sklearn.metrics import f1 score, roc auc score, precision recall fscore support y true = 0, 1, 1, 0, 1 y score = 0.2, 0.9, 0.7, 0.1, 0.6 threshold = 0.5 y pred = 1 if s = threshold else 0 for s in y score f1 = f1 score y true, y pred auc = roc auc score y true, y score p, r, , = precision recall fscore support y true, y pred, average="binary" print f"F1={f1:.3f} AUC={auc:.3f} precision={p:.3f} recall={r:.3f}" For multi-class routing, use per-class F1 and an averaging choice you can defend against how errors cost you. More on this topic, with the related episodes, is on Chain of Thought https://chainofthought.show/topics/evaluation/ . It draws on this episode https://chainofthought.show/podcast/57-every-ai-agent-has-an-evaluation-gap-alex-ratner-snorkel-ai/ . Subscribe to the Chain of Thought newsletter https://newsletter.chainofthought.show/subscribe?utm source=devto&utm medium=referral&utm campaign=newsletter for new episodes and write-ups like this one. Drafted with AI assistance from the episode transcripts.