Accuracy is a lie when you're dealing with imbalanced datasets in binary classification. I hit a wall recently where my model showed 92% accuracy during training, but the precision-recall curve looked like a disaster. The core issue is the fundamental difference between the objective function (what the optimizer minimizes) and the evaluation metrics (what we actually care about).
For anyone building a custom training loop, don't rely on the raw loss. I use
In PyTorch, we typically use BCEWithLogitsLoss
for binary tasks. This is a smooth, differentiable function necessary for backpropagation. However, you can't use accuracy as a loss function because it's non-differentiable.
To properly diagnose model performance, I've moved to a setup that tracks these specific metrics per epoch:
Log Loss (Cross-Entropy): The actual value the optimizer is attacking. If this drops but accuracy doesn't move, you're likely dealing with a severe class imbalance.Precision & Recall: Essential for understanding the trade-off between false positives and false negatives.F1-Score: The harmonic mean of the two, which is the only number I trust for "overall" performance on skewed data.AUC-ROC: Measures the model's ability to distinguish between classes regardless of the decision threshold.
For anyone building a custom training loop, don't rely on the raw loss. I use
torchmetrics
to handle the accumulation across batches to avoid the "average of averages" mistake. Here is the basic logic for calculating the F1-score correctly within a validation loop:
import torch
from torchmetrics.classification import BinaryF1Score
f1_metric = BinaryF1Score().to(device)
model.eval()
with torch.no_grad():
for inputs, targets in val_:
inputs, targets = inputs.to(device), targets.to(device)
logits = model(inputs)
preds = (torch.sigmoid(logits) > 0.5).float()
f1_metric.update(preds, targets)
final_f1 = f1_metric.compute()
print(f"Validation F1 Score: {final_f1}")
If you see your loss decreasing while your F1-score plateaus or drops, check your class weights in the loss function. Adding pos_weight
to BCEWithLogitsLoss
usually fixes the bias toward the majority class.
Next The Stack v3: Massive Scale vs. Data Noise →
All Replies （3） #
R
Happened to me with fraud detection; high accuracy but it missed every single actual case.
0
S
Did you try adjusting the class weights in BCE loss, or just pray it worked?
0
P
Using an F1-score usually clears things up for me when the classes are skewed.
0