PyTorch Binary Classification: Fixing the Loss vs. Metric Gap

A developer describes a common pitfall in PyTorch binary classification where high accuracy (92%) on imbalanced datasets masks poor precision-recall performance, recommending tracking metrics like F1-score and AUC-ROC via torchmetrics instead of relying solely on loss. The post advises using class weights in BCEWithLogitsLoss to address class imbalance and provides code for computing validation F1-score correctly.

PyTorch Binary Classification: Fixing the Loss vs. Metric Gap Accuracy is a lie when you're dealing with imbalanced datasets in binary classification. I hit a wall recently where my model showed 92% accuracy during training, but the precision-recall curve looked like a disaster. The core issue is the fundamental difference between the objective function what the optimizer minimizes and the evaluation metrics what we actually care about . For anyone building a custom training loop, don't rely on the raw loss. I use In PyTorch, we typically use BCEWithLogitsLoss for binary tasks. This is a smooth, differentiable function necessary for backpropagation. However, you can't use accuracy as a loss function because it's non-differentiable. To properly diagnose model performance, I've moved to a setup that tracks these specific metrics per epoch: Log Loss Cross-Entropy : The actual value the optimizer is attacking. If this drops but accuracy doesn't move, you're likely dealing with a severe class imbalance. Precision & Recall: Essential for understanding the trade-off between false positives and false negatives. F1-Score: The harmonic mean of the two, which is the only number I trust for "overall" performance on skewed data. AUC-ROC: Measures the model's ability to distinguish between classes regardless of the decision threshold. For anyone building a custom training loop, don't rely on the raw loss. I use torchmetrics to handle the accumulation across batches to avoid the "average of averages" mistake. Here is the basic logic for calculating the F1-score correctly within a validation loop: python import torch from torchmetrics.classification import BinaryF1Score Initialize metric f1 metric = BinaryF1Score .to device model.eval with torch.no grad : for inputs, targets in val loader: inputs, targets = inputs.to device , targets.to device logits = model inputs Convert logits to binary predictions 0 or 1 preds = torch.sigmoid logits 0.5 .float f1 metric.update preds, targets final f1 = f1 metric.compute print f"Validation F1 Score: {final f1}" If you see your loss decreasing while your F1-score plateaus or drops, check your class weights in the loss function. Adding pos weight to BCEWithLogitsLoss usually fixes the bias toward the majority class. Next The Stack v3: Massive Scale vs. Data Noise → /en/threads/2755/ All Replies （3） R Happened to me with fraud detection; high accuracy but it missed every single actual case. 0 S Did you try adjusting the class weights in BCE loss, or just pray it worked? 0 P Using an F1-score usually clears things up for me when the classes are skewed. 0