Gaussian Naive Bayes From Scratch — The computation behind every prediction

A tutorial builds Gaussian Naive Bayes from scratch and compares it against scikit-learn's version on the Wisconsin Breast Cancer dataset, which has 569 samples and 30 numerical features. The algorithm fits a normal distribution to each feature per class, computing means and variances during training, then uses the Gaussian log-likelihood for predictions. The piece explains how the curved decision boundaries arise from unequal class variances.

Most machine learning tutorials hand you a model.fit call and move on. This one goes the other direction — we're going to build Gaussian Naive Bayes ourselves, run it against scikit-learn's version on the breast cancer dataset, and then look at what the learning curves actually reveal about how this algorithm thinks. Let’s check a image first to get the idea of Naive Bayes. The scatter plot above gives you an intuition for what Gaussian Naive Bayes actually produces. Each colored cluster represents a class — and notice the curved lines separating them. Unlike a straight-line boundary you’d get from logistic regression, Gaussian NB draws these smooth curves because each class gets its own fitted bell curve with its own spread variance . Where those bell curves intersect is where the boundary falls. The more unequal the spreads between classes, the more curved the boundary becomes. You might already know the basic Naive Bayes idea which uses Bayes’ Theorem to compute the probability of each class, multiply by some per-feature likelihoods, and pick the winner. That works perfectly when your features are categorical — like weather labels Sunny/Rainy/Overcast in the classic Play Tennis example. But what happens when your features are continuous numbers — like tumor measurements in millimetres? You can’t just count how many training samples had mean radius = 14.23. That exact value might never repeat. Gaussian Naive Bayes solves this by fitting a normal distribution bell curve to each feature per class. Once you have that fitted curve, you can plug in any value and get back a probability density. That density is then used as the likelihood, and the rest of the algorithm proceeds exactly as before. The log-likelihood of a single value x given a Gaussian with mean μ and variance σ² is: log N x | μ, σ² = -0.5 × log 2π σ² - x - μ ² / 2σ² Two things to notice here : first, the exponent contains x - μ ² — the squared distance from the mean. Values close to the class mean score high; values far away score low. Second, we work in log-space throughout, which turns all those tiny multiplications into additions and avoids floating-point underflow. We’re using the Wisconsin Breast Cancer dataset, which comes built into scikit-learn. It has 569 samples , each described by 30 numerical features — things like the mean radius, texture, and smoothness of a cell nucleus. The label is binary: malignant 0 or benign 1 . python from sklearn.datasets import load breast cancerfrom sklearn.model selection import train test split X, y = load breast cancer return X y=True X train, X test, y train, y test = train test split X, y, test size=0.2, random state=42 455 training samples, 114 test samples During training, Gaussian Naive Bayes doesn’t optimize anything. It just computes three things for each class: python import numpy as np python class GaussianNB: def fit self, X, y : X, y = np.asarray X , np.asarray y self. classes = np.unique y n features = X.shape 1 n classes = len self. classes self. mean = np.zeros n classes, n features self. variance = np.zeros n classes, n features self. priors = np.zeros n classes for i, c in enumerate self. classes : X class = X y == c self. mean i = X class.mean axis=0 self. variance i = X class.var axis=0 self. priors i = X class.shape 0 / X.shape 0 return self After fit, we have a 2 × 30 table of means and another of variances — that's literally the entire trained model. No weights, no loss function, no epochs. At prediction time, for each test sample, we need to compute how well it fits the distribution of each class. This is where the Gaussian formula comes in: python def log gaussian self, x : x shape: n samples, n features self. mean shape: n classes, n features diff = x :, None, : - self. mean → n samples, n classes, n features num = - diff 2 / 2 self. variance log prob = num - 0.5 np.log 2 np.pi self. variance return log prob.sum axis=2 → n samples, n classes The x :, None, : trick inserts a class axis so NumPy can broadcast the subtraction cleanly across all classes at once — no loops needed. After summing across features, we get one log-likelihood score per sample per class. python def predict self, X : X = np.asarray X log likelihood = self.log gaussian X log prior = np.log self. priors return self. classes np.argmax log likelihood + log prior, axis=1 Adding log prior in log-space, this is addition, not multiplication shifts the scores to account for class imbalance before we take the argmax. The class with the highest combined score wins. My model accuracy : 96.49%Sklearn accuracy : 97.37%Prediction agreement: 99.12% Two things stand out here. First, the accuracy gap between our implementation and sklearn’s is tiny — just under 1%. For 114 test samples, that’s literally one prediction different. Second, and more interestingly the two models agreed on 99.1% of individual predictions. They’re effectively the same algorithm. The marginal difference traces back to sklearn applying a small var smoothing constant 1e-9 that stabilizes variance estimates on very flat features. Our implementation doesn't do that, which occasionally nudges one borderline sample the other way. This is where things get genuinely interesting. A learning curve shows how accuracy changes as you give the model more training data — and the shape of that curve tells you things about the algorithm that a single accuracy number never could. Learning Curve — Custom Implementation A few things jump out immediately - The test accuracy flatlines at 96.5% from around 150 samples onwards. This is a signature behaviour of Naive Bayes — it’s a generative model, so it just needs enough samples to estimate its Gaussians reliably. Once those estimates stabilize, adding more data barely changes anything. The early instability 50–150 samples comes from variance estimates being unreliable with small samples. With only 50 rows, one unusual sample can wildly shift the estimated spread of a feature. Learning Curve — Scikit-Learn The sklearn curve is smoother overall, but tells the same fundamental story. The test ceiling is marginally higher 97.3% , and the train curve is less volatile, largely due to the variance-smoothing constant doing quite stabilizing work in the background. One notable difference - sklearn’s test accuracy actually rises slightly at the far right rather than flattening completely. This is because with more training data, the variance estimates themselves become better, which tightens the Gaussian fits and nudges a few borderline predictions the right way. If you look at a learning curve for a decision tree or a neural network, you’d expect to see train accuracy stay high while test accuracy slowly climbs to meet it — the classic overfitting-to-generalization convergence. Naive Bayes doesn’t really do that. Because it makes a strong structural assumption features are independent given the class, and each follows a Gaussian , it can’t “overfit” in the traditional sense — there’s no complexity to crank up. What it can do is have unspecified assumptions — the Gaussians might not be the right shape for some features. More data helps, but only up to the point where the estimates are stable. After that, the ceiling is set by the quality of the assumptions, not the quantity of data. This is precisely why Naive Bayes is often described as a high-bias, low-variance model. It has a ceiling it can’t break through without changing the model family — but it reaches that ceiling fast, and it’s stable once it gets there. python import numpy as npfrom sklearn.datasets import load breast cancerfrom sklearn.metrics import accuracy scorefrom sklearn.model selection import train test splitfrom sklearn.naive bayes import GaussianNB as SkGaussianNBimport plotly.graph objects as go python class GaussianNB: def fit self, X, y : X, y = np.asarray X , np.asarray y self. classes = np.unique y n features = X.shape 1 n classes = len self. classes self. mean = np.zeros n classes, n features self. variance = np.zeros n classes, n features self. priors = np.zeros n classes for i, c in enumerate self. classes : X class = X y == c self. mean i = X class.mean axis=0 self. variance i = X class.var axis=0 self. priors i = X class.shape 0 / X.shape 0 return self def log gaussian self, x : diff = x :, None, : - self. mean num = - diff 2 / 2 self. variance log prob = num - 0.5 np.log 2 np.pi self. variance return log prob.sum axis=2 def predict self, X : X = np.asarray X log likelihood = self.log gaussian X log prior = np.log self. priors return self. classes np.argmax log likelihood + log prior, axis=1 X, y = load breast cancer return X y=True X train, X test, y train, y test = train test split X, y, test size=0.2, random state=42 nb classifier = GaussianNB .fit X train, y train my pred = nb classifier.predict X test print f"My model accuracy : {accuracy score my pred, y test :.4f}" sk nb = SkGaussianNB sk nb.fit X train, y train sk pred = sk nb.predict X test print f"Sklearn accuracy : {accuracy score sk pred, y test :.4f}" print f"Prediction agreement: {np.mean my pred == sk pred :.4f}" Learning curvestrain sizes = np.linspace 0.1, 1.0, 10 my train scores, my test scores = , sk train scores, sk test scores = , for frac in train sizes: n = int frac len X train X sub, y sub = X train :n , y train :n my model = GaussianNB .fit X sub, y sub my train scores.append accuracy score y sub, my model.predict X sub my test scores.append accuracy score y test, my model.predict X test sk model = SkGaussianNB .fit X sub, y sub sk train scores.append accuracy score y sub, sk model.predict X sub sk test scores.append accuracy score y test, sk model.predict X test x axis = train sizes len X train .astype int for title, train s, test s in "Learning Curve - My Gaussian Naive Bayes", my train scores, my test scores , "Learning Curve - Sklearn Gaussian Naive Bayes", sk train scores, sk test scores , : fig = go.Figure fig.add trace go.Scatter x=x axis, y=train s, mode="lines+markers", name="Train Accuracy" fig.add trace go.Scatter x=x axis, y=test s, mode="lines+markers", name="Test Accuracy" fig.update layout title=title, xaxis title="Training Samples", yaxis title="Accuracy", template="plotly white" fig.show Now try swapping the breast cancer dataset for a different one and watching how the learning curve shape changes. Code for this implementation is on my GitHub — feel free to fork it and try the next round of improvements yourself. https://github.com/Archan47/Machine-Learning-Algorithms-From-Scratch Gaussian Naive Bayes From Scratch — The computation behind every prediction https://pub.towardsai.net/gaussian-naive-bayes-from-scratch-the-computation-behind-every-prediction-ccfaffb7ceff was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.