CatBoost: The Interpreter Who Refused to Peek at Tomorrow's Newspaper

A developer's blog post explains how CatBoost's ordered target statistics prevent target leakage in categorical feature encoding, a common issue in tabular machine learning. The post illustrates the problem with an analogy of court interpreters who, given the case file including the verdict, appear skilled on old cases but are merely leaking future information. CatBoost's method ensures each row only sees data from preceding rows, avoiding the leak that can make training loss misleadingly low and test loss five times worse.

The One-Line Summary:Replacing a category with the average target for that category is the most common feature-engineering trick in tabular machine learning and it quietly hands the model the answer — measured here, it turned a column containingliterally no signalinto a training loss of 0.2466 and a test loss of 2.7975, five times worse than deleting the column; CatBoost's ordered target statistics fix the leak by letting each row see only rows that came before it. The court employed interpreters, because testimony arrived in six languages and the judges read one. An interpreter's job was to render each witness's words faithfully, and for two centuries the court considered this a solved problem. Then someone noticed that the interpreters were extraordinarily good at old cases and merely average at new ones. Every interpreter was handed the complete case file before beginning. This was considered basic professionalism — how could you translate testimony about a boundary dispute without knowing it was a boundary dispute? The file included the verdict. WHY THE OLD INTERPRETERS LOOKED SO GOOD ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Witness says a word that could mean "he took it" or "he was given it". Interpreter has read the file. The file says GUILTY. She writes "he took it." She is not lying. She is not even conscious of choosing. The ambiguity resolved itself the moment she knew how it ended. On closed cases her renderings are uncanny. On open cases she is ordinary, and nobody can work out why. The interpreters were not corrupt. They were contaminated , which is worse, because contamination leaves no one to blame and nothing obvious to fix. A clerk named Ilaya was asked to audit the interpreters and did something nobody had tried: she took a hundred old cases, stripped the verdicts, and had them re-translated by the same people. THE SAME INTERPRETERS, VERDICTS HIDDEN ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ with the file as always near-perfect renderings without the verdict ordinary renderings indistinguishable from a first-year apprentice The skill everyone had been admiring for two centuries was not skill. It was the verdict, travelling backwards into the testimony. The court's instinct was to ban case files entirely. Ilaya argued against it: context genuinely helps, and an interpreter working blind is worse than one working informed. The problem was never the file. It was which parts of the file . "Let her read everything that was written before the words she is translating. Nothing that was written after. The rule is not ignorance — it is chronology." ILAYA'S RULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Translating testimony from the 3rd of the month? You may read: everything filed on the 1st, 2nd. You may not read: the 4th onward. Ever. Especially the verdict. Each document is translated using only the court's knowledge AS IT STOOD at that moment. Slower. Occasionally the interpreter has almost no context and must simply admit it. Those renderings are honest, and they hold up. The first interpreters trained this way looked worse on the court's historical records. They were the first ones who did not get worse when the case was live. Hard pivot. You have a categorical column with many levels — city, merchant, user ID, product SKU. One-hot encoding it produces thousands of columns. The popular alternative is target encoding : replace each level with the mean of the target for that level. Look at the index set. It includes j=i . Row i 's own label is in the numerator of row i 's feature. If a level appears once, its encoding is its label, exactly. If it appears three times, the encoding is two-thirds someone else's answer and one-third your own. That is the verdict travelling backwards into the testimony. CatBoost computes ordered target statistics . Fix a random permutation σ of the rows. For row i , use only rows that precede it: where p is a prior the global mean and a its weight. Row i is excluded by construction — it hasn't happened yet. The prior does the work when history is thin. The same principle is applied to the gradients during boosting, which is where the name "ordered boosting" comes from, but the target-statistic version is where the damage usually is. The cleanest possible test: build a categorical column that carries no information whatsoever , and see whether target encoding can conjure signal out of it. 6,000 rows, 2,000 levels — roughly three rows per level, which is exactly the regime where people reach for target encoding. python import numpy as np, pandas as pd from sklearn.model selection import train test split, KFold from sklearn.metrics import log loss from sklearn.ensemble import HistGradientBoostingClassifier rng = np.random.default rng 0 n, K = 6000, 2000 cat = rng.integers 0, K, n pure noise: 2,000 levels x1 = rng.normal size=n y = 0.9 x1 + rng.normal 0, 1.0, n 0 .astype int y ignores cat X = pd.DataFrame {"cat": cat, "x1": x1} Xtr, Xte, ytr, yte = train test split X, y, test size=0.3, random state=0 ctr, cte = Xtr "cat" .values, Xte "cat" .values gm = ytr.mean def score A, B : m = HistGradientBoostingClassifier max iter=200, random state=0 .fit A, ytr return log loss ytr, m.predict proba A :, 1 , log loss yte, m.predict proba B :, 1 full mean = pd.Series ytr .groupby ctr .mean B = Xte.copy ; B "cat" = pd.Series cte .map full mean .fillna gm .values A = Xtr.copy ; A "cat" = pd.Series ctr .map full mean .fillna gm .values tr, te = score A, B print f" naive target encoding train {tr:.4f} test {te:.4f} gap {te-tr:+.4f}" A2 = Xtr.copy ; enc = np.full len Xtr , np.nan ; ys = pd.Series ytr for itr, iva in KFold 5, shuffle=True, random state=0 .split Xtr : mm = ys.iloc itr .groupby ctr itr .mean enc iva = pd.Series ctr iva .map mm .fillna ys.iloc itr .mean .values A2 "cat" = enc tr, te = score A2, B print f" out-of-fold encoding train {tr:.4f} test {te:.4f} gap {te-tr:+.4f}" tr, te = score Xtr "x1" , Xte "x1" print f" drop the column train {tr:.4f} test {te:.4f} gap {te-tr:+.4f}" naive target encoding train 0.2466 test 2.7975 gap +2.5508 out-of-fold encoding train 0.4436 test 0.5823 gap +0.1388 drop the column train 0.4964 test 0.5464 gap +0.0500 Sit with the first row. The column is random integers . It has no relationship to the target — I generated y from x1 alone. And naive target encoding produced a training loss of 0.2466 , better than the honest model can achieve, and a test loss of 2.7975 — roughly five times worse than simply deleting the column. A model that has memorised the training set and learned nothing looks, during training, exactly like a breakthrough. Ilaya's rule is about thirty lines. Walk the rows in a random permutation, encode each one from the running totals so far, then add it to the totals. python import numpy as np def ordered target stats cat, y, prior, a=1.0, seed=0 : """Each row is encoded using only rows earlier in the permutation.""" rng = np.random.default rng seed perm = rng.permutation len cat enc = np.empty len cat csum, ccnt = {}, {} for pos in perm: c = cat pos s, k = csum.get c, 0.0 , ccnt.get c, 0 enc pos = s + a prior / k + a history only — self excluded csum c = s + y pos now add self, for later rows ccnt c = k + 1 return enc The two lines after the assignment are the whole idea. The row contributes to everyone after it and never to itself. Run all four encodings on the same data, both when the column is noise and when it genuinely carries signal, three seeds each: === NO signal in the column test logloss, 3 seeds === seed naive oof ordered drop 0 2.7975 0.5823 0.5812 0.5464 1 3.2548 0.5780 0.6050 0.5545 2 2.9780 0.5988 0.6014 0.5702 MEAN 3.0101 0.5863 0.5959 0.5570 === REAL signal in the column test logloss, 3 seeds === seed naive oof ordered drop 0 2.2485 0.5707 0.6737 0.6398 1 2.5480 0.5564 0.6239 0.6139 2 2.0538 0.5601 0.6670 0.6518 MEAN 2.2834 0.5624 0.6549 0.6352 Four readings, and the third one surprised me. Naive encoding is a catastrophe either way — 3.0101 and 2.2834 against roughly 0.56 for every honest method. It does not matter whether the column has signal. The leak dominates. When the column carries real signal, encoding it pays. Out-of-fold scored 0.5624 against 0.6352 for dropping the column. That is the case for target encoding, and it is a real one. My ordered implementation lost to plain out-of-fold — 0.6549 against 0.5624. I expected the opposite, and I am reporting it because it is what the code printed. Dropping a signal-free column beats every encoding of it. 0.5570 against 0.5863 out-of-fold. No scheme recovers information that was never there; encoding only adds noise with a plausible face. A result you did not expect is worth ten minutes before you publish it. The obvious suspect is variance: early rows in the permutation have almost no history, so their encodings are mostly prior. CatBoost's actual answer to this is to average several permutations — so I tried that. python import numpy as np, pandas as pd from sklearn.model selection import train test split from sklearn.metrics import log loss from sklearn.ensemble import HistGradientBoostingClassifier def ordered target stats cat, y, prior, a=1.0, seed=0 : rng = np.random.default rng seed perm = rng.permutation len cat enc = np.empty len cat ; csum, ccnt = {}, {} for pos in perm: c = cat pos ; s, k = csum.get c, 0.0 , ccnt.get c, 0 enc pos = s + a prior / k + a csum c = s + y pos ; ccnt c = k + 1 return enc rng = np.random.default rng 0 n, K = 6000, 2000 cat = rng.integers 0, K, n eff = rng.normal 0, 1.5, K this time the column MATTERS x1 = rng.normal size=n y = eff cat + 0.9 x1 + rng.normal 0, 1.0, n 0 .astype int X = pd.DataFrame {"cat": cat, "x1": x1} Xtr, Xte, ytr, yte = train test split X, y, test size=0.3, random state=0 ctr, cte = Xtr "cat" .values, Xte "cat" .values gm = ytr.mean fm = pd.Series ytr .groupby ctr .mean B = Xte.copy ; B "cat" = pd.Series cte .map fm .fillna gm .values print " permutations test logloss" for P in 1, 2, 4, 8, 16 : encs = np.mean ordered target stats ctr, ytr, gm, seed=s for s in range P , axis=0 A = Xtr.copy ; A "cat" = encs m = HistGradientBoostingClassifier max iter=200, random state=0 .fit A, ytr print f" {P: 12} {log loss yte, m.predict proba B :, 1 :.4f}" and the diagnostic that explains the result perm = np.random.default rng 0 .permutation len ctr seen, hist = {}, for pos in perm: c = ctr pos ; hist.append seen.get c, 0 ; seen c = seen.get c, 0 + 1 hist = np.array hist print f"\n rows with ZERO prior observations of their own level: " f"{ hist==0 .sum :,} of {len hist :,} { hist==0 .mean 100:.1f}% " print f" first 10% of the permutation: " f"{ hist :len hist //10 ==0 .mean 100:.1f}% have no history" print f" last 10% of the permutation: " f"{ hist -len hist //10: ==0 .mean 100:.1f}% have no history" permutations test logloss 1 0.6737 2 0.7020 4 0.7110 8 0.6810 16 0.6854 Averaging did not help. So the problem is not variance, and I went looking for what it actually is: rows with ZERO prior observations of their own level: 1,747 of 4,200 41.6% first 10% of the permutation: 92.1% have no history last 10% of the permutation: 13.1% have no history 41.6% of training rows never see a single earlier example of their own category. At 2,000 levels across 4,200 training rows, most levels appear two or three times, so for a large fraction of rows there is simply nothing legitimate to learn from. Their encoding is the global prior — a constant. Averaging permutations cannot manufacture history that does not exist. Out-of-fold encoding wins here because it is less strict: a row in fold 3 gets the mean over folds 1, 2, 4 and 5 — around 80% of the data rather than "whatever happened to precede me". Both exclude the row's own label, which is the part that matters. Ordered TS excludes more, and at this cardinality it excludes too much. One caveat I want to be explicit about: this is my thirty-line reimplementation of the principle, not CatBoost. Real CatBoost recomputes target statistics inside the boosting loop with a fresh permutation per tree, builds combinations of categorical features, and applies the same ordering idea to gradients. Do not read this table as "CatBoost loses to out-of-fold encoding" — read it as "the ordering principle, applied naively as a preprocessing step, runs out of history at extreme cardinality." TARGET ENCODING: A DECISION PROCEDURE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Does the column plausibly carry signal? | +- No - DROP IT. Measured best in every | no-signal run. Encoding a useless | column only invents noise. | +- Yes / unsure | +- Fewer than ~50 levels? | - one-hot. Boring, safe, competitive. | +- Many levels, several rows each? | - out-of-fold target encoding. | Best measured result here. | +- Many levels, 1-3 rows each? - the column is nearly an ID. Expect little. Consider dropping, or hashing to fewer buckets. NEVER: mean over all training rows including self. train 0.2466 / test 2.7975 on a column that contained nothing. The tell that you have leaked is always the same shape: training loss far better than anything the problem should allow, and a validation gap that widens as you add capacity. A gap of +2.55 log loss is not overfitting. Overfitting is +0.05. That is contamination. | CatBoost | LightGBM | XGBoost | | |---|---|---|---| | Categorical handling | ordered target statistics, built in | native partition or one-hot | you encode it yourself | | Leak protection | by construction | none — your job | none — your job | | Tree shape | oblivious same split per level | leaf-wise | depth-wise | | Defaults on small data | conservative | dangerous num leaves=31 | safe | | Best fit | many high-cardinality categoricals | large numeric tabular | general tabular | Reach for CatBoost when your table is mostly categorical with real cardinality — retail SKUs, merchant IDs, geography — and you would otherwise be hand-rolling encodings and getting them subtly wrong. Reach elsewhere when your features are numeric, where its advantage largely disappears. CATBOOST / TARGET ENCODING: CHEAT SHEET ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ THE LEAK: x i = mean y j for j where c j == c i includes j == i. Row i's own label is inside row i's feature. One row per level = the encoding IS the label. THE FIX ordered target statistics : fix a permutation; for row i use only rows BEFORE it: x i = sum of earlier y + a prior / count of earlier + a self excluded by construction. MEASURED column with NO signal at all : naive train 0.2466 test 2.7975 out-of-fold train 0.4436 test 0.5823 drop it train 0.4964 test 0.5464 <- best MEASURED column WITH signal : naive 2.2834 | oof 0.5624 | ordered 0.6549 | drop 0.6352 oof wins WHY ORDERED CAN LOSE: 41.6% of rows had NO earlier row of their own level 2,000 levels / 4,200 rows . Averaging 16 permutations did not help — the history isn't thin, it's absent. THE TELL YOU LEAKED: train loss impossibly good + gap that GROWS with capacity. Overfitting is +0.05. Leakage was +2.55. Naive target encoding leaks the label into the feature. The row's own y sits in the numerator of its own x . It manufactures signal from nothing. On a column of random integers: train 0.2466, test 2.7975 — five times worse than deleting the column. Ordered target statistics fix it by chronology, not ignorance. Each row sees only rows earlier in a random permutation; it is excluded from its own encoding by construction. Out-of-fold encoding was the best real method measured — 0.5624 against 0.6352 for dropping a signal-bearing column, and it beat my ordered implementation. Dropping a signal-free column beats every encoding of it 0.5570 vs 0.5863 . Encoding cannot recover information that was never present. Ordering can exclude too much. 41.6% of rows had no earlier example of their own level, and averaging 16 permutations did not help — the fix for absent history is not more permutations. Leakage and overfitting look different. Overfitting widened the gap by 0.05. Leakage widened it by 2.55. If your training metric looks impossible, it is. Check your own reimplementation before blaming the library. My thirty-line ordered TS is not CatBoost, and I said so rather than reporting a benchmark I had not actually run. Target encoding is the most widely used trick in tabular machine learning and its textbook form quietly puts each row's own answer inside that row's features — CatBoost's contribution is Ilaya's rule, that a row may be described using everything the data knew before it and nothing the data learned after, which is not a clever optimisation but a refusal to look at tomorrow's newspaper while reporting today's news. Follow me for the next article in the Boosting: The Complete Guide series If the court at Vashti made target leakage click, drop a heart Questions? Ask in the comments — I read and respond to every one. Have you ever shipped a target-encoded feature without checking the fold boundaries? I have, and the model was the best I had ever built right up until the day it met data it hadn't already been told the answer to. ⚖️ The thing that unsettles me about leakage is that it never feels like cheating while you are doing it. Every step is defensible — use the data you have, summarise the category, feed the model context. The failure is assembled entirely from reasonable decisions, which is why code review does not catch it and a suspiciously good training curve does. Learn to distrust your best results specifically because they are your best results.