# Kernels, margins, and ensembles

> Source: <https://stochastic.blog/kernels-margins-and-ensembles/>
> Published: 2026-09-04 11:12:26+00:00

[Stochastic Blog](https://stochastic.blog/tag/stochastic-blog/)

# Kernels, margins, and ensembles

In Posts 11 through 13 we built classifiers that draw boundaries directly, fitting logistic curves and trees to the Adult Census Income data. This post takes the opposite route and then the long way around: we start with **generative classifiers** that model how each income class produces its features, pivot to **Support Vector Machines** that hunt for the widest possible margin between classes, and finish by asking whether combining many weak models can beat any single strong one. The through-line is a simple question we keep asking at every step: what does this model actually see when it looks at a row of census data? By the end we have an answer measured by AUC, the area under the receiver operating characteristic curve, which tells how often the model ranks a random above-50K row above a random at-or-below-50K row. 0.5 is chance, 1.0 is perfect. The best single number lands at 0.907.

The dataset is the UCI Adult Census Income table, 32,561 rows after download, each describing a person's age, education, occupation, and weekly hours against a binary label of whether they earn over 50K. After dropping the 24 duplicate rows, the cleaned frame has 32,537 rows. From that cleaned frame we take a stratified 25 percent sample of 8,134 rows, meaning the sample preserves the same 75.9/24.1 class split as the full data, and we hold out 1,627 rows for every evaluation that follows.

## What the data says first

Before any model sees a row, we need an honest picture of what we are predicting. The label balance is the first finding that shapes everything: 75.9 percent of rows earn at most 50K, so a model that predicts the majority class for every row scores 0.759 accuracy. Any classifier worth reporting has to clear that floor. The second finding is where the missing values live. Only three categorical fields, workclass, occupation, and native_country, contain gaps, and we fill them with an explicit Unknown category rather than dropping rows. The third finding is quieter but matters for the ensembles later: capital_gain and capital_loss are heavily right skewed with most values at zero, while hours_per_week clusters near 40 with a long tail of longer weeks.

The histograms show the skew we expected, but the correlation matrix tells a cleaner story. The numeric features barely correlate with each other, which means the signal for high income has to come from combinations of features rather than any single one. That observation will return when we reach the kernel methods.

We also note that education_num is an ordinal coding of education, so we model with it alone and leave the redundant education string out.

## Generative classifiers

**Generative classifiers** take a different stance from the discriminative models we built before. Instead of drawing a boundary directly, they model the probability of the features given each class, then invert with Bayes rule to get the class probability. We test five of them: **Gaussian Naive Bayes**, **Multinomial Naive Bayes**, **Bernoulli Naive Bayes**, **Linear Discriminant Analysis**, and **Quadratic Discriminant Analysis**. The contrast between discriminative and generative approaches is the real lesson: discriminative models ask where the boundary lies, generative models ask how each class produces its data.

The implementation is a single evaluation loop that fits each model and records accuracy, AUC, and fit time. The preprocessing pipeline scales the five numeric features to [0, 1] and one-hot encodes the seven categorical fields, turning each category into its own 0/1 column, producing 90 features total. We define the five model instances that the loop will compare as follows.

```
generative_models = [
    ('gaussian_naive_bayes', GaussianNB()),
    ('multinomial_naive_bayes', MultinomialNB()),
    ('bernoulli_naive_bayes', BernoulliNB(binarize=0.5)),
    ('linear_discriminant_analysis', LinearDiscriminantAnalysis()),
    ('quadratic_discriminant_analysis', QuadraticDiscriminantAnalysis(reg_param=0.01))
]
```

The results split cleanly. Gaussian Naive Bayes collapses to 0.316 accuracy with an AUC of 0.620, because its Gaussian assumption on the skewed capital fields is badly wrong. Multinomial Naive Bayes recovers to 0.763 accuracy and 0.854 AUC, and Bernoulli Naive Bayes lands at 0.738 accuracy and 0.840 AUC. Linear Discriminant Analysis is the strongest generative model at 0.822 accuracy and 0.878 AUC, while Quadratic Discriminant Analysis trails at 0.735 accuracy and 0.854 AUC. The pattern is clear: the naive Bayes models pay for conditional independence, LDA's pooled covariance makes it the strongest generative model, and QDA's per-class covariance overtunes and trails.

## Margins and kernels

The pivot to **Support Vector Machines** changes the question. Instead of modeling how each class generates features, an SVM finds the **maximum-margin classifier**, the boundary that sits as far as possible from the nearest training points. The distance to those nearest points is the margin, and the points that touch it are the support vectors. A hard margin that lets no point cross is brittle, so we allow **soft margins** where points can violate the boundary at a price controlled by the parameter C. The cost of crossing is the **hinge loss**, which grows linearly with how far a point overshoots the margin.

We can see the soft margin tradeoff on synthetic blobs. With C set to 0.05 the boundary tolerates many violations and stays smooth; with C at 10.0 the model chases nearly every point and the boundary contorts.

The hinge loss on this toy data averages 0.01, meaning almost every point sits comfortably on the correct side. But linear boundaries fail on interleaved data, which is where the **kernel trick** enters. An **RBF kernel** and a **polynomial kernel** compute dot products in an implicit high-dimensional feature space without ever constructing that space. The moons dataset makes the difference visible: the linear kernel draws a straight line through two interleaved crescents, while the RBF and polynomial kernels wrap around the structure.

**Mercer's condition** tells us whether a candidate kernel is valid: the Gram matrix of pairwise similarities must be positive semidefinite, meaning all of its eigenvalues are nonnegative. We check both kernels on a 30 point slice and confirm the smallest eigenvalues are 0.0 for RBF and -0.0 for polynomial, both safely above the numerical tolerance.

On the real census data, SVM training cost grows superlinearly, so we train on a 2,000 row stratified slice. The linear SVM reaches 0.841 accuracy and 0.888 AUC, while the RBF SVM lands at 0.816 accuracy and 0.859 AUC. The linear model wins here, which is a useful reminder that kernel flexibility is not free. On tabular data with 90 mostly sparse features, the linear boundary captures most of the signal without the risk of overfitting the margin.

## Ensembles

A single boundary is limited, so the ensemble question is whether many weak learners can beat it. The baseline comparison is a single depth-3 decision tree, which reaches 0.840 accuracy and 0.845 AUC. **Bagging** trains many bootstrap trees and averages their votes, and 80 such trees lift AUC to 0.868. **Random Forests** add **feature subsampling**, where each split sees only a random subset of features, and the difference is immediate. With max_features set to sqrt, the forest reaches 0.844 accuracy and 0.886 AUC, while using all features drops to 0.837 accuracy and 0.881 AUC. The **out-of-bag error** gives a free validation score from rows each tree never saw. To reproduce the out-of-bag score, configure the forest as follows:

```
rf_oob = RandomForestClassifier(
    n_estimators=100,
    max_features='sqrt',
    oob_score=True,
    n_jobs=-1,
    random_state=SEED
)
```

That configuration reports an out-of-bag accuracy of 0.842, close to the held-out accuracy.

Boosting takes the opposite strategy. **AdaBoost** reweights misclassified samples and minimizes an **exponential loss**, and 60 shallow trees reach 0.845 accuracy and 0.896 AUC. The exponential loss on the test set is 0.689, a number that matters because it is the quantity AdaBoost actually minimizes. **Gradient boosting** builds trees that predict residuals rather than reweighting samples, and 80 depth 2 trees reach 0.851 accuracy and 0.902 AUC. The modern implementations push further: **XGBoost** hits 0.856 accuracy and 0.907 AUC, **LightGBM** matches that AUC at 0.907 with 0.857 accuracy, and **CatBoost** lands at 0.853 accuracy and 0.904 AUC.

The final move is **stacking**, which trains a meta model on cross-validated base model predictions, and **blending**, the same idea with a single holdout split. We stack a Gaussian Naive Bayes, a linear SVM, and a gradient boosted tree, letting a logistic regression learn how to combine their outputs.

```
base_estimators = [
    ('gaussian_nb', GaussianNB()),
    ('linear_svm', LinearSVC(C=1.0, max_iter=5000, random_state=SEED)),
    ('gradient_boosting', GradientBoostingClassifier(n_estimators=80, max_depth=2, random_state=SEED))
]
```

The stacked model reaches 0.852 accuracy and 0.901 AUC, and the blended version scores 0.854 accuracy and 0.900 AUC. Neither beats the best single gradient boosted model, which is the honest result. Stacking does not always outperform its strongest base model, but it flattens variance and adds a fraction of AUC when the base models are diverse. The Gaussian Naive Bayes is the weakest model in the stack, yet its class probabilities still supply signal the SVM and the boosted tree miss, which is why the stack holds its own.

The full ranking puts LightGBM and XGBoost at the top with 0.907 AUC, followed by CatBoost at 0.904, gradient boosting at 0.902, and the stacked model at 0.901. The generative models trail, and the single tree sits at the bottom of the ensemble family.

## Closing

The EDA findings predicted this outcome. The 75.9-to-24.1 class imbalance meant accuracy alone was a weak yardstick, so we tracked AUC throughout. The skewed capital fields punished Gaussian Naive Bayes and rewarded tree methods that split on thresholds. The low correlations between numeric features meant interactions mattered, which is exactly what boosted trees capture by construction. Every modeling choice traced back to something we saw in the first figures.

The notebook simplifies in ways a production system would not. We train on a 25 percent sample to keep CPU time under 30 minutes, we use a single train test split rather than cross-validation for the final numbers, and we do not tune hyperparameters beyond the defaults and a few hand picked values. A production version would sweep C for the SVM, search tree depth and learning rate jointly, and validate with repeated stratified folds.

The question this post leaves open is whether the 0.907 AUC ceiling comes from the model family or from the feature representation. The next post in the series moves from tabular classifiers to gradient boosted models at scale, where the answer depends on how far engineering can push the same algorithms.

The exercises in the notebook let you test the edges of what we built. Swap Bernoulli Naive Bayes thresholds to 0.25 and 0.75 and watch the AUC move. Train a polynomial kernel SVM on the same 2,000 row slice and compare fit time with the RBF version. Change the random forest max_features from sqrt to log2 and report the out-of-bag shift. Build a stacking ensemble without the SVM and compare AUC with the full three model stack. Add education and fnlwgt to the feature list and find where extra cardinality starts to hurt.

## Further reading

by Christopher Bishop. Chapter 4 covers linear classifiers and generative models with the mathematical care this post skates over.*Pattern Recognition and Machine Learning*by Trevor Hastie, Robert Tibshirani, and Jerome Friedman. Chapter 4 develops linear discriminative methods, and later chapters give the theory of boosting and random forests.*The Elements of Statistical Learning*by Kevin Murphy. Chapter 12 treats generative classifiers in depth and connects them to the discriminative alternatives.*Probabilistic Machine Learning: An Introduction*