You can ship a polished eval dashboard and still learn nothing about whether the product works. A common way to get there is picking a number because it looked authoritative somewhere else, then discovering that production behavior and the chart tell different stories. Before you wire graphs, name the feature shape (classifier, ranker, generator, or grounded answerer) and choose a metric that matches that shape.
If your LLM feature ends in a yes or no (my examples: toxic or not, escalate or not, intent A vs B), you have a classification problem even when the model reasons in natural language. For imbalanced labels, accuracy misleads: a model that always predicts the majority class can score high while catching none of the cases you care about. The F1 score is the harmonic mean of precision and recall, so it stays low if either side is weak. A system cannot game F1 by maximizing precision while missing every positive, or by calling everything positive to inflate recall.
I still split precision and recall when I need to know which failure mode hurts users. F1 weights the two equally; your error costs might not.
When the model outputs a score and you have not fixed the decision threshold, AUC-ROC is a reasonable summary for comparing models on the same data. It reflects how well positives rank above negatives as you sweep thresholds. AUC near 0.5 is random separation on that set; near 1.0 is strong separation.
On rare positives (fraud, abuse, defects), AUC can look strong while most positive predictions are still wrong, because a small false positive rate on a huge negative pool still produces many false alarms. When positives are rare, report precision and recall, or F1 at the threshold you plan to ship, alongside AUC.
Autoregressive models assign probabilities to next tokens. Perplexity rolls that into one number: lower means the held-out text surprised the model less. It helps during training and when you compare models on the same unseen text with the same tokenizer. Change the tokenizer or let test data leak into training, and the comparison breaks.
Perplexity is not well defined for masked language models the way it is for standard autoregressive LMs, and it measures prediction on text, not success on your task. Better perplexity does not guarantee better outcomes on the work you ship. A model can assign high probability to fluent continuations and still hallucinate, ignore instructions, or fail your rubric. I use perplexity as a development signal for like-for-like model comparison, not as proof a chat, copilot, or agent is ready for users.
Generators need task evals: rubrics, human review, judges where that is appropriate, and checks tied to the user request. Only measuring the application end to end tells you whether the feature works.
RAG and citation-backed answers raise a different question: did the model stay inside the retrieved context, or did it invent details? Faithfulness (sometimes called groundedness) checks whether claims in the answer are supported by the supplied sources. It targets hallucination in the sense of adding material that is not in the context.
Faithfulness is separate from whether the answer is correct. An answer can be fully faithful to the wrong source documents. Pair it with context relevance (was the right material retrieved?) and answer relevance (did the model address the question?), not as a single headline KPI.
If you want your model to do something different, you don't go and perform brain surgery on the neurons. You don't reinvent the architecture of the algorithm. Let me go and invent a new post-transformer architecture. No, you get data.
Alex Ratner, Snorkel AI, on Chain of Thought ep 57
Ratner's point is that data is the primary tool for changing a model, even if everything matters. Measurement data works the same way: pick the instrument to match the feature, then invest in labels and rubrics that expose gaps before users do.
For binary classifier features, I start with a labelled holdout set that reflects production skew. Choose a threshold on a validation slice using the false positive and false negative costs you accept, then report F1 at that threshold for the story stakeholders will feel. Keep AUC on scored examples to compare models before you lock a cutoff.
Minimal Python sketch (not production code):
from sklearn.metrics import f1_score, roc_auc_score, precision_recall_fscore_support
y_true = [0, 1, 1, 0, 1]
y_score = [0.2, 0.9, 0.7, 0.1, 0.6]
threshold = 0.5
y_pred = [1 if s >= threshold else 0 for s in y_score]
f1 = f1_score(y_true, y_pred)
auc = roc_auc_score(y_true, y_score)
p, r, _, _ = precision_recall_fscore_support(y_true, y_pred, average="binary")
print(f"F1={f1:.3f} AUC={auc:.3f} precision={p:.3f} recall={r:.3f}")
For multi-class routing, use per-class F1 and an averaging choice you can defend against how errors cost you.
More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.