How to Navigate the Bias-Variance Tradeoff and Double Descent Ethically in Machine Learning

In machine learning, we’ve always lived by the bias-variance tradeoff, a constant balancing act, and more recently, the more nuanced phenomenon of double descent.

In this post, I’ll break down what these tradeoffs mean, why they matter, and, most importantly, how to navigate them ethically when building and deploying real-world ML systems.

Bias happens when a model is too simple, it misses crucial patterns in the data, leading to flawed, predictable errors. It is called underfitting.

On the other hand, variance occurs when a model is too complex, it memorizes every tiny, irrelevant detail (noise) specific to the training data. This makes it perform poorly on new information, which we call overfitting.

The tradeoff:

The goal is to find the perfect middle ground, the sweet spot: a model complex enough to be accurate but simple enough to generalize. Too simple, and it misses important patterns. Too complex, and it overreacts to every little noise in the data.

Modern deep learning has added an unexpected twist to the story — something called double descent, introduced by Belkin at al. (2019).

According to the classic bias-variance tradeoff, there’s a sweet spot for model complexity. Make the model too simple, and it underfits. Make it too complex, and performance takes a nosedive because of overfitting.

But double descent flips that idea. After a model hits its worst point, where it’s clearly overfitting, something strange happens. If you keep making it more complex, instead of getting worse, the model actually starts to improve again. Its ability to handle new data gets better, not worse.

Understanding these ideas and how model complexity really works isn’t just a technical detail. It directly shapes the choices we make about which models to use, and those choices can have real-world, ethical consequences. For example:

When a model is too complex, it doesn’t just learn the patterns, it can also learn from the noise and hidden biases in the training data. If our data reflects societal inequalities, such as racial bias in loan approvals, an overfit model might reinforce those biases in ways that are not immediately obvious.

Suggestion: Use disparity metrics, test to evaluate how our model performs across various attributes such as race, gender, …

A high-bias model tends to oversimplify things, it can miss important differences in the data, especially if it’s working with limited features or a narrow view of the problem.

For example, if we use a model that’s too simple in healthcare, it might treat similar symptoms the same way across all patients, without accounting for how those symptoms can show up differently in different populations, and lead to misdiagnoses. Suggestion: Include domain experts in model design to catch oversimplified assumptions early, regularly evaluate performance across different subgroups to ensure the model isn’t missing patterns that matter.

When we enter into the world of overparameterized models, while performance often gets better, our understanding of how the model makes its decisions becomes much harder. On one hand, these models can deliver more accurate results, which sounds great. But on the other hand, their lack of transparency means it’s harder to spot when something goes wrong or to explain why it went wrong in the first place.

Suggestion: Use explainability tools such as [SHAP ](http://Bowen, Dillon, and Lyle Ungar. "Generalized SHAP: Generating multiple types of explanations in machine learning." arXiv preprint arXiv:2006.07155 (2020).)and [LIME](http://Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. "" Why should i trust you?" Explaining the predictions of any classifier." In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144. 2016.).

Here’s a simple practical framework for making ethical decisions when facing these tradeoffs:

Ask these questions: Who could be impacted by this model?, What are we trading off? (accuracy vs transparency vs complexity vs fairness etc.), Can we explain the decisions?, Have we tested across real-world scenarios and groups?

2. Contextual evaluation

Understand the domain; collaborate with domain experts during model design and when interpreting results.

3. Representation-Aware Training

Analyze our training data for bias.

4. Transparency in Complexity

When deploying a high-capacity model, such as in the second descent, accompany it with explainability tools (such as SHAP and LIME) or risk mitigation strategies.

5. Iterative Feedback Loops

Establish mechanisms to monitor and update models over time.

References:

Elton, Daniel C. (2020). “Self-explaining AI as an Alternative to Interpretable AI.” In: Goertzel, B., Panov, A., Potapov, A., Yampolskiy, R. (eds) Artificial General Intelligence. AGI 2020. Lecture Notes in Computer Science(), vol 12177. Springer, Cham. https://doi.org/10.1007/978-3-030-52152-3_10

McCarthy, Michael B., Narayanan, Sundaraparipurnan. 2023. “Fairness–accuracy tradeoff: activation function choice in a neural network.” AI Ethics 3 1423–1432.

Prendin, Francesco, Pavan, Jacopo, Cappon, Giacomo. et al. 2023. “The importance of interpreting machine learning models for blood glucose prediction in diabetes: an analysis using SHAP”. Scientific Reports 13, 16865.

Eunjin Lee, David Braines, Mitchell Stiffler, Adam Hudler, and Daniel Harborne. 2019. “Developing the sensitivity of LIME for better machine learning explanation”, Proc. SPIE 11006, Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, 1100610.

LIME vs. SHAP: Which is Better for Explaining Machine Learning Models? | Towards Data Science

LinkedIn: https://www.linkedin.com/in/selinkarabulut/ How to Navigate the Bias-Variance Tradeoff and Double Descent Ethically in Machine Learning was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.