Feature Selection Techniques: More Features Does Not Mean a Better Model

Feature selection is a critical but underused step in machine learning, as irrelevant features waste compute, invite overfitting, and hide signal behind noise, according to a technical article. The article categorizes techniques into filter, wrapper, and embedded methods, noting that filter methods like Pearson correlation can miss non-linear relationships, while mutual information captures any statistical dependency. It emphasizes that choosing the right filter is essential for real-world, non-linear data.

There is a developer on every team who feeds the model every column in the database, because more information has to be better. Then the training run triples in length, the test set score drops, and nobody can explain why the model behaves the way it does. This is the dimensionality trap , and it is not a beginner mistake. It is the default behavior of anyone who treats feature selection as optional. Every dataset contains features that matter and features that do not. The irrelevant ones do not sit quietly; they waste compute, invite overfitting, and hide the signal behind noise. The fix is not a better model. The fix is a smaller, sharper set of inputs. Feature selection is the process of identifying which variables actually contribute to prediction and removing the rest. Done well, it improves accuracy, speeds up training, and makes models dramatically easier to interpret. It is also one of the most underused techniques in applied machine learning, because it sounds like a preprocessing detail when it is actually a modeling decision. Before the techniques, the economics. An irrelevant feature is not a free passenger; it has four distinct costs. Feature selection is the process of identifying which variables actually contribute to prediction and removing the rest, so that the model learns from signal instead of noise. Every feature selection technique belongs to one of three families, and the family determines what the method can and cannot see. Filter methods score each feature independently using statistical tests. They are fast and model-agnostic but blind to interactions: a feature that is useless on its own but powerful in combination with another will be discarded before it gets a chance. Wrapper methods train a model on different feature subsets and keep the best, capturing interactions at the price of serious compute. Embedded methods perform selection during model training itself, striking the balance between speed and accuracy that most production systems need. Filter methods rank features by how strongly they relate to the target, without ever training a model. That makes them the cheapest possible first pass, and on high-dimensional data they are often the only thing that can run at all. The choice of statistic depends on your data types: Every filter has the same blind spot: it looks at each feature in isolation. A feature that only matters in combination with another will rank low no matter how powerful the combination is. That is acceptable when the filter is a first pass. It is fatal when the filter is the whole strategy. The difference between Pearson correlation and mutual information is where filter methods earn or lose their reputation. Pearson correlation only detects linear relationships. Point it at a feature with a strong U-shaped relationship to the target, and it will report a score near zero, even though the feature is one of the most informative in the dataset. Mutual information measures how much knowing a feature reduces uncertainty about the target. It does not care whether the relationship is linear, quadratic, or something stranger; any statistical dependency counts. This is the property that matters in real data, because real relationships are rarely straight lines. A customer’s churn risk can peak in the middle of the tenure range, a user’s engagement can rise then fall with session count, and none of that shows up in a correlation coefficient. """filter methods comparison.pyFilter methods: why correlation can miss real signal.Builds a dataset where the target depends on x1 linearly, on x2quadratically a U-shape , and not at all on x3, then compares howPearson correlation and mutual information rank the three features.Expected result: Pearson scores the U-shaped feature near zero,indistinguishable from pure noise. Mutual information ranks it as oneof the strongest features in the dataset. This is why choosing theright filter matters on real, non-linear data."""import numpy as npfrom sklearn.feature selection import f regression, mutual info regressionrng = np.random.default rng 42 n = 2 000x1 = rng.uniform -1, 1, n linear driverx2 = rng.uniform -1, 1, n quadratic U-shaped driverx3 = rng.uniform -1, 1, n pure noisey = 2.0 x1 + 4.0 x2 2 + rng.normal 0, 0.15, n features = {"x1 linear ": x1, "x2 U-shaped ": x2, "x3 noise ": x3}print f"{'Feature':<15} {'Pearson r': 10} {'F-test': 8} {'Mutual info': 12}" print "-" 50 for name, x in features.items : r = np.corrcoef x, y 0, 1 f stat, = f regression x.reshape -1, 1 , y mi = mutual info regression x.reshape -1, 1 , y, random state=42 0 print f"{name:<15} {r: 10.3f} {f stat 0 : 8.1f} {mi: 12.4f}" print "\nPearson correlation misses the U-shaped relationship r ~ 0.00 " print "because it only measures linear dependence." print "Mutual information detects it, because it measures any dependence." The snippet builds the exact failure case: a target driven linearly by one feature, quadratically by a second, and not at all by a third. Pearson correlation ranks the quadratic feature near zero, indistinguishable from pure noise. Mutual information ranks it correctly as one of the strongest features in the dataset. Run it and you have seen the entire argument for choosing the right filter, in numbers. The practical rule: if you know your relationships are linear, Pearson is faster and perfectly adequate. If you do not know, or you have any reason to suspect curvature, use mutual information. “When in doubt, trust mutual information” is a good default for tabular data, and the compute penalty is modest for a first pass. Wrapper methods treat feature selection as a search problem: instead of scoring features from the outside, they train a model repeatedly and let its performance decide which features stay. The most common strategies are: Recursive Feature Elimination RFE trains a model on all features, ranks them by importance, drops the weakest, and retrains on the remainder, repeating until the target number of features remains. It is the wrapper workhorse because it works with any model that exposes feature importances, from logistic regression to gradient boosters. Forward selection starts with an empty set and adds the feature that improves performance most, one at a time, until no addition helps. Backward elimination starts with everything and removes the least useful feature each round, until removal stops helping. Because wrappers use the actual model, they capture interactions that filters cannot see. That is their power, and also their weakness. Every retrain multiplies compute, so wrapper methods are impractical on large feature sets. And because they optimize on the same data they search, they are the most prone to overfitting: a wrapper can happily select features that fit the noise of the training set. Always wrap the search in cross-validation, and use the production model you will actually ship, not a stand-in. RFE has one practical knob worth knowing. You can specify the number of features to keep, but you do not have to guess it: cross-validated RFE tests performance at each subset size and picks the number automatically. That is the difference between “let me keep ten features” and “let the data decide how many features earn their keep.” Embedded methods bake feature selection into model training itself. You train once, and selection arrives as a side effect of the training process, with no separate preprocessing step. There are two families of embedded selection, and both are everyday tools. Regularization methods add a penalty to the model’s coefficients that shrinks them toward zero. Lasso applies an L1 penalty that drives some coefficients to exactly zero, which is selection by construction: features with zero coefficients are not features anymore. Elastic Net blends L1 and L2 penalties, keeping the selection behavior of Lasso while handling groups of correlated features more gracefully. Tree-based importance measures how much each feature reduces impurity across all the splits of a random forest or gradient booster. A feature used in many clean splits scores high; a feature that never splits anything scores near zero. This is the practical default for most tabular problems: it is fast, it handles non-linear interactions natively, and it requires no assumptions about the distribution of your data. """embedded lasso selection.pyEmbedded methods: Lasso selects features by shrinking coefficients toexactly zero during training.Builds synthetic regression data with 10 features, only 4 of which areinformative, then fits Lasso L1 regularization and inspects thecoefficients. Informative features keep meaningful weights; noisefeatures shrink to exactly zero, which is feature selection byconstruction. No separate preprocessing step needed."""import numpy as npfrom sklearn.datasets import make regressionfrom sklearn.linear model import LassoCVfrom sklearn.preprocessing import StandardScalerX, y = make regression n samples=500, n features=10, n informative=4, noise=0.5, random state=42, Standardize so the L1 penalty treats every feature fairly.X scaled = StandardScaler .fit transform X LassoCV picks the regularization strength by cross-validation.lasso = LassoCV cv=5, random state=42 lasso.fit X scaled, y print f"Selected alpha: {lasso.alpha :.4f}\n" print f"{'Feature':<10} {'Coefficient': 12} Selected" print "-" 40 for i, coef in enumerate lasso.coef : selected = "yes" if abs coef 1e-6 else "no" print f"feature {i + 1:<4} {coef: 12.4f} {selected}" n selected = int np.sum abs lasso.coef 1e-6 print f"\n{n selected} of 10 features survived Lasso selection." The snippet runs Lasso on synthetic data with only four informative features hiding among ten. The printed coefficients make the selection visible: the informative features keep meaningful weights while the noise features shrink to exactly zero. That is the entire point of embedded selection, demonstrated in one table of numbers. The known weakness of tree importance is correlated features. When two features carry the same signal, the trees distribute importance between them, so each one reports a lower score than it deserves. The features are not wrong; the ranking is shared. The same dilution happens with regularization, where correlated features split a coefficient between them. This matters when you use importance to rank features: rank, then investigate, and do not treat the scores as independent evidence about each feature. In practice, the best results come from combining the families, because each one covers the blind spot of the others. Filters are fast but blind to interactions. Wrappers capture interactions but are slow. Embedded methods are in between. A layered pipeline uses each for what it is best at: """feature selection pipeline.pyThe hybrid pipeline: filter, then embed, then wrap.1. Filter: SelectKBest with mutual information cuts the raw set.2. Wrapper: RFE with a random forest refines it to the final k.3. Model: The production classifier trains on the survivors.Also compares cross-validated performance of the model trained on allfeatures against the model trained on the selected subset, so thepayoff of selection is visible in one run."""import timeimport numpy as npimport pandas as pdfrom sklearn.datasets import make classificationfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.feature selection import SelectKBest, mutual info classif, RFEfrom sklearn.model selection import cross val score, train test splitfrom sklearn.pipeline import Pipeline 60 features: 8 informative, 2 redundant, 50 pure noise.X, y = make classification n samples=1500, n features=60, n informative=8, n redundant=2, random state=42, X = pd.DataFrame X, columns= f"feature {i + 1}" for i in range 60 X train, X test, y train, y test = train test split X, y, test size=0.2, stratify=y, random state=42 Filter: select top features by mutual informationfilter step = SelectKBest mutual info classif, k=20 Embedded + Wrapper: RFE with Random Forestselector = RFE estimator=RandomForestClassifier n estimators=100, random state=42 , n features to select=10 Combined pipelinepipeline = Pipeline "filter", filter step , "rfe", selector , "model", RandomForestClassifier n estimators=100, random state=42 , Baseline: the same model on all 30 features, for comparison.baseline = RandomForestClassifier n estimators=100, random state=42 Cross-validated AUC for both, on the training portion.cv selected = cross val score pipeline, X train, y train, cv=5, scoring="roc auc" cv all = cross val score baseline, X train, y train, cv=5, scoring="roc auc" Wall-clock training time for one fit of each, on the full training set.t0 = time.perf counter baseline.fit X train, y train time all = time.perf counter - t0t0 = time.perf counter pipeline.fit X train, y train time selected = time.perf counter - t0print f"CV ROC-AUC, all 60 features : {cv all.mean :.4f}" print f"CV ROC-AUC, selected 10 : {cv selected.mean :.4f}" print f"Fit time, baseline : {time all:.2f}s" print f"Fit time, hybrid + selection : {time selected:.2f}s" print "\nSelection is a one-time cost: the hybrid pipeline pays it once," print "then every retrain and every inference runs on 10 features" print "instead of 60." Map the selection masks back to the original column names.filter mask = pipeline.named steps "filter" .get support rfe mask = pipeline.named steps "rfe" .support selected = X train.columns filter mask rfe mask print f"\nSelected features {len selected } :" print list selected The snippet is the hybrid pipeline in production form: a mutual information filter cuts the feature set, RFE with a random forest refines it, and the final model trains on what survives. It also reports the payoff, comparing the model’s cross-validated performance on the full feature set against the selected subset: on the synthetic data it builds, ten selected features beat all sixty, because the fifty noise columns were dragging the model down. The selection itself is a one-time cost; every retrain and every inference after it runs on ten features instead of sixty. This layered approach has a quality that the individual methods lack on their own: robustness. Each stage narrows the search space for the next one, so the expensive wrapper runs on a small, pre-cleaned candidate set instead of the raw haystack. The result is selection that is fast enough to iterate on and accurate enough to trust. The right family depends on three constraints, and it is worth being honest about all three before starting. The only genuinely wrong choice is using all of your features by default. Whatever the family, the selection step itself is never optional. It is a modeling decision, made before the model is trained, and it shapes everything the model can and cannot learn. Feature selection is not a tuning trick. It is a modeling decision. Filter methods give you speed. Wrappers give you accuracy. Embedded methods give you both. The right choice depends on your data size, your compute budget, and whether you need to understand why each feature matters, but the direction is always the same: fewer, sharper features in, better model out. A smaller feature set is a sharper model. Start with a filter, thin with an embedded method, finish with the wrapper if accuracy demands it, and let the data justify every column that survives. Here are several key takeaways from this article: Thank you for reading this article I hope you found it helpful. If you have any questions or feedback, please feel free to reach out to me. Feature Selection Techniques: More Features Does Not Mean a Better Model https://pub.towardsai.net/feature-selection-techniques-more-features-does-not-mean-a-better-model-33ba863b3c44 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.