The One-Line Summary:Replacing a category with the average target for that category is the most common feature-engineering trick in tabular machine learning and it quietly hands the model the answer β measured here, it turned a column containingliterally no signalinto a training loss of 0.2466 and a test loss of 2.7975, five times worse than deleting the column; CatBoost's ordered target statistics fix the leak by letting each row see only rows that came before it.
The court employed interpreters, because testimony arrived in six languages and the judges read one. An interpreter's job was to render each witness's words faithfully, and for two centuries the court considered this a solved problem.
Then someone noticed that the interpreters were extraordinarily good at old cases and merely average at new ones.
Every interpreter was handed the complete case file before beginning. This was considered basic professionalism β how could you translate testimony about a boundary dispute without knowing it was a boundary dispute?
The file included the verdict.
WHY THE OLD INTERPRETERS LOOKED SO GOOD
ββββββββββββββββββββββββββββββββββββββββββββββββββ
Witness says a word that could mean
"he took it" or "he was given it".
Interpreter has read the file. The file says
GUILTY.
She writes "he took it."
She is not lying. She is not even conscious of
choosing. The ambiguity resolved itself the
moment she knew how it ended.
On closed cases her renderings are uncanny.
On open cases she is ordinary, and nobody can
work out why.
The interpreters were not corrupt. They were contaminated, which is worse, because contamination leaves no one to blame and nothing obvious to fix.
A clerk named Ilaya was asked to audit the interpreters and did something nobody had tried: she took a hundred old cases, stripped the verdicts, and had them re-translated by the same people.
THE SAME INTERPRETERS, VERDICTS HIDDEN
ββββββββββββββββββββββββββββββββββββββββββββββββββ
with the file (as always) near-perfect renderings
without the verdict ordinary renderings
indistinguishable from
a first-year apprentice
The skill everyone had been admiring for two
centuries was not skill. It was the verdict,
travelling backwards into the testimony.
The court's instinct was to ban case files entirely. Ilaya argued against it: context genuinely helps, and an interpreter working blind is worse than one working informed. The problem was never the file. It was which parts of the file.
"Let her read everything that was written before the words she is translating. Nothing that was written after. The rule is not ignorance β it is chronology."
ILAYA'S RULE
ββββββββββββββββββββββββββββββββββββββββββββββββββ
Translating testimony from the 3rd of the month?
You may read: everything filed on the 1st, 2nd.
You may not read: the 4th onward. Ever.
Especially the verdict.
Each document is translated using only the
court's knowledge AS IT STOOD at that moment.
Slower. Occasionally the interpreter has almost
no context and must simply admit it.
Those renderings are honest, and they hold up.
The first interpreters trained this way looked worse on the court's historical records. They were the first ones who did not get worse when the case was live.
Hard pivot.
You have a categorical column with many levels β city, merchant, user ID, product SKU. One-hot encoding it produces thousands of columns. The popular alternative is target encoding: replace each level with the mean of the target for that level.
Look at the index set. It includes j=i . Row i 's own label is in the numerator of row i 's feature. If a level appears once, its encoding is its label, exactly. If it appears three times, the encoding is two-thirds someone else's answer and one-third your own.
That is the verdict travelling backwards into the testimony.
CatBoost computes ordered target statistics. Fix a random permutation Ο of the rows. For row i , use only rows that precede it:
where p is a prior (the global mean) and a its weight. Row i is excluded by construction β it hasn't happened yet. The prior does the work when history is thin.
The same principle is applied to the gradients during boosting, which is where the name "ordered boosting" comes from, but the target-statistic version is where the damage usually is.
The cleanest possible test: build a categorical column that carries no information whatsoever, and see whether target encoding can conjure signal out of it. 6,000 rows, 2,000 levels β roughly three rows per level, which is exactly the regime where people reach for target encoding.
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split, KFold
from sklearn.metrics import log_loss
from sklearn.ensemble import HistGradientBoostingClassifier
rng = np.random.default_rng(0)
n, K = 6000, 2000
cat = rng.integers(0, K, n) # pure noise: 2,000 levels
x1 = rng.normal(size=n)
y = (0.9 * x1 + rng.normal(0, 1.0, n) > 0).astype(int) # y ignores cat
X = pd.DataFrame({"cat": cat, "x1": x1})
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
ctr, cte = Xtr["cat"].values, Xte["cat"].values
gm = ytr.mean()
def score(A, B):
m = HistGradientBoostingClassifier(max_iter=200, random_state=0).fit(A, ytr)
return (log_loss(ytr, m.predict_proba(A)[:, 1]),
log_loss(yte, m.predict_proba(B)[:, 1]))
full_mean = pd.Series(ytr).groupby(ctr).mean()
B = Xte.copy(); B["cat"] = pd.Series(cte).map(full_mean).fillna(gm).values
A = Xtr.copy(); A["cat"] = pd.Series(ctr).map(full_mean).fillna(gm).values
tr, te = score(A, B)
print(f" naive target encoding train {tr:.4f} test {te:.4f} gap {te-tr:+.4f}")
A2 = Xtr.copy(); enc = np.full(len(Xtr), np.nan); ys = pd.Series(ytr)
for itr, iva in KFold(5, shuffle=True, random_state=0).split(Xtr):
mm = ys.iloc[itr].groupby(ctr[itr]).mean()
enc[iva] = pd.Series(ctr[iva]).map(mm).fillna(ys.iloc[itr].mean()).values
A2["cat"] = enc
tr, te = score(A2, B)
print(f" out-of-fold encoding train {tr:.4f} test {te:.4f} gap {te-tr:+.4f}")
tr, te = score(Xtr[["x1"]], Xte[["x1"]])
print(f" drop the column train {tr:.4f} test {te:.4f} gap {te-tr:+.4f}")
naive target encoding train 0.2466 test 2.7975 gap +2.5508
out-of-fold encoding train 0.4436 test 0.5823 gap +0.1388
drop the column train 0.4964 test 0.5464 gap +0.0500
Sit with the first row.
The column is random integers. It has no relationship to the target β I generated y
from x1
alone. And naive target encoding produced a training loss of 0.2466, better than the honest model can achieve, and a test loss of 2.7975 β roughly five times worse than simply deleting the column.
A model that has memorised the training set and learned nothing looks, during training, exactly like a breakthrough.
Ilaya's rule is about thirty lines. Walk the rows in a random permutation, encode each one from the running totals so far, then add it to the totals.
import numpy as np
def ordered_target_stats(cat, y, prior, a=1.0, seed=0):
"""Each row is encoded using only rows earlier in the permutation."""
rng = np.random.default_rng(seed)
perm = rng.permutation(len(cat))
enc = np.empty(len(cat))
csum, ccnt = {}, {}
for pos in perm:
c = cat[pos]
s, k = csum.get(c, 0.0), ccnt.get(c, 0)
enc[pos] = (s + a * prior) / (k + a) # history only β self excluded
csum[c] = s + y[pos] # now add self, for later rows
ccnt[c] = k + 1
return enc
The two lines after the assignment are the whole idea. The row contributes to everyone after it and never to itself.
Run all four encodings on the same data, both when the column is noise and when it genuinely carries signal, three seeds each:
=== NO signal in the column (test logloss, 3 seeds) ===
seed naive oof ordered drop
0 2.7975 0.5823 0.5812 0.5464
1 3.2548 0.5780 0.6050 0.5545
2 2.9780 0.5988 0.6014 0.5702
MEAN 3.0101 0.5863 0.5959 0.5570
=== REAL signal in the column (test logloss, 3 seeds) ===
seed naive oof ordered drop
0 2.2485 0.5707 0.6737 0.6398
1 2.5480 0.5564 0.6239 0.6139
2 2.0538 0.5601 0.6670 0.6518
MEAN 2.2834 0.5624 0.6549 0.6352
Four readings, and the third one surprised me.
Naive encoding is a catastrophe either way β 3.0101 and 2.2834 against roughly 0.56 for every honest method. It does not matter whether the column has signal. The leak dominates.
When the column carries real signal, encoding it pays. Out-of-fold scored 0.5624 against 0.6352 for dropping the column. That is the case for target encoding, and it is a real one.
My ordered implementation lost to plain out-of-fold β 0.6549 against 0.5624. I expected the opposite, and I am reporting it because it is what the code printed.
Dropping a signal-free column beats every encoding of it. 0.5570 against 0.5863 out-of-fold. No scheme recovers information that was never there; encoding only adds noise with a plausible face.
A result you did not expect is worth ten minutes before you publish it. The obvious suspect is variance: early rows in the permutation have almost no history, so their encodings are mostly prior. CatBoost's actual answer to this is to average several permutations β so I tried that.
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss
from sklearn.ensemble import HistGradientBoostingClassifier
def ordered_target_stats(cat, y, prior, a=1.0, seed=0):
rng = np.random.default_rng(seed)
perm = rng.permutation(len(cat))
enc = np.empty(len(cat)); csum, ccnt = {}, {}
for pos in perm:
c = cat[pos]; s, k = csum.get(c, 0.0), ccnt.get(c, 0)
enc[pos] = (s + a * prior) / (k + a)
csum[c] = s + y[pos]; ccnt[c] = k + 1
return enc
rng = np.random.default_rng(0)
n, K = 6000, 2000
cat = rng.integers(0, K, n)
eff = rng.normal(0, 1.5, K) # this time the column MATTERS
x1 = rng.normal(size=n)
y = (eff[cat] + 0.9 * x1 + rng.normal(0, 1.0, n) > 0).astype(int)
X = pd.DataFrame({"cat": cat, "x1": x1})
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
ctr, cte = Xtr["cat"].values, Xte["cat"].values
gm = ytr.mean()
fm = pd.Series(ytr).groupby(ctr).mean()
B = Xte.copy(); B["cat"] = pd.Series(cte).map(fm).fillna(gm).values
print(" permutations test logloss")
for P in (1, 2, 4, 8, 16):
encs = np.mean([ordered_target_stats(ctr, ytr, gm, seed=s)
for s in range(P)], axis=0)
A = Xtr.copy(); A["cat"] = encs
m = HistGradientBoostingClassifier(max_iter=200, random_state=0).fit(A, ytr)
print(f" {P:>12} {log_loss(yte, m.predict_proba(B)[:, 1]):.4f}")
perm = np.random.default_rng(0).permutation(len(ctr))
seen, hist = {}, []
for pos in perm:
c = ctr[pos]; hist.append(seen.get(c, 0)); seen[c] = seen.get(c, 0) + 1
hist = np.array(hist)
print(f"\n rows with ZERO prior observations of their own level: "
f"{(hist==0).sum():,} of {len(hist):,} ({(hist==0).mean()*100:.1f}%)")
print(f" first 10% of the permutation: "
f"{(hist[:len(hist)//10]==0).mean()*100:.1f}% have no history")
print(f" last 10% of the permutation: "
f"{(hist[-len(hist)//10:]==0).mean()*100:.1f}% have no history")
permutations test logloss
1 0.6737
2 0.7020
4 0.7110
8 0.6810
16 0.6854
Averaging did not help. So the problem is not variance, and I went looking for what it actually is:
rows with ZERO prior observations of their own level: 1,747 of 4,200 (41.6%)
first 10% of the permutation: 92.1% have no history
last 10% of the permutation: 13.1% have no history
41.6% of training rows never see a single earlier example of their own category. At 2,000 levels across 4,200 training rows, most levels appear two or three times, so for a large fraction of rows there is simply nothing legitimate to learn from. Their encoding is the global prior β a constant. Averaging permutations cannot manufacture history that does not exist.
Out-of-fold encoding wins here because it is less strict: a row in fold 3 gets the mean over folds 1, 2, 4 and 5 β around 80% of the data rather than "whatever happened to precede me". Both exclude the row's own label, which is the part that matters. Ordered TS excludes more, and at this cardinality it excludes too much.
One caveat I want to be explicit about: this is my thirty-line reimplementation of the principle, not CatBoost. Real CatBoost recomputes target statistics inside the boosting loop with a fresh permutation per tree, builds combinations of categorical features, and applies the same ordering idea to gradients. Do not read this table as "CatBoost loses to out-of-fold encoding" β read it as "the ordering principle, applied naively as a preprocessing step, runs out of history at extreme cardinality."
TARGET ENCODING: A DECISION PROCEDURE
ββββββββββββββββββββββββββββββββββββββββββββββββββ
Does the column plausibly carry signal?
|
+- No -> DROP IT. Measured best in every
| no-signal run. Encoding a useless
| column only invents noise.
|
+- Yes / unsure
|
+- Fewer than ~50 levels?
| -> one-hot. Boring, safe, competitive.
|
+- Many levels, several rows each?
| -> out-of-fold target encoding.
| Best measured result here.
|
+- Many levels, 1-3 rows each?
-> the column is nearly an ID.
Expect little. Consider dropping,
or hashing to fewer buckets.
NEVER: mean over all training rows including self.
train 0.2466 / test 2.7975 on a column
that contained nothing.
The tell that you have leaked is always the same shape: training loss far better than anything the problem should allow, and a validation gap that widens as you add capacity. A gap of +2.55 log loss is not overfitting. Overfitting is +0.05. That is contamination.
| CatBoost | LightGBM | XGBoost | |
|---|---|---|---|
| Categorical handling | ordered target statistics, built in | native partition or one-hot | you encode it yourself |
| Leak protection | by construction | none β your job | none β your job |
| Tree shape | oblivious (same split per level) | leaf-wise | depth-wise |
| Defaults on small data | conservative | dangerous (num_leaves=31 ) |
|
| safe | |||
| Best fit | many high-cardinality categoricals | large numeric tabular | general tabular |
Reach for CatBoost when your table is mostly categorical with real cardinality β retail SKUs, merchant IDs, geography β and you would otherwise be hand-rolling encodings and getting them subtly wrong. Reach elsewhere when your features are numeric, where its advantage largely disappears.
CATBOOST / TARGET ENCODING: CHEAT SHEET
ββββββββββββββββββββββββββββββββββββββββββββββββββ
THE LEAK:
x_i = mean(y_j for j where c_j == c_i)
includes j == i. Row i's own label is inside
row i's feature. One row per level => the
encoding IS the label.
THE FIX (ordered target statistics):
fix a permutation; for row i use only rows
BEFORE it:
x_i = (sum of earlier y + a*prior)
/ (count of earlier + a)
self excluded by construction.
MEASURED (column with NO signal at all):
naive train 0.2466 test 2.7975
out-of-fold train 0.4436 test 0.5823
drop it train 0.4964 test 0.5464 <- best
MEASURED (column WITH signal):
naive 2.2834 | oof 0.5624 | ordered 0.6549
| drop 0.6352 oof wins
WHY ORDERED CAN LOSE:
41.6% of rows had NO earlier row of their own
level (2,000 levels / 4,200 rows). Averaging
16 permutations did not help β the history
isn't thin, it's absent.
THE TELL YOU LEAKED:
train loss impossibly good + gap that GROWS
with capacity. Overfitting is +0.05.
Leakage was +2.55.
Naive target encoding leaks the label into the feature. The row's own y sits in the numerator of its own x .
It manufactures signal from nothing. On a column of random integers: train 0.2466, test 2.7975 β five times worse than deleting the column.
Ordered target statistics fix it by chronology, not ignorance. Each row sees only rows earlier in a random permutation; it is excluded from its own encoding by construction.
Out-of-fold encoding was the best real method measured β 0.5624 against 0.6352 for dropping a signal-bearing column, and it beat my ordered implementation.
Dropping a signal-free column beats every encoding of it (0.5570 vs 0.5863). Encoding cannot recover information that was never present.
Ordering can exclude too much. 41.6% of rows had no earlier example of their own level, and averaging 16 permutations did not help β the fix for absent history is not more permutations.
Leakage and overfitting look different. Overfitting widened the gap by 0.05. Leakage widened it by 2.55. If your training metric looks impossible, it is.
Check your own reimplementation before blaming the library. My thirty-line ordered TS is not CatBoost, and I said so rather than reporting a benchmark I had not actually run.
Target encoding is the most widely used trick in tabular machine learning and its textbook form quietly puts each row's own answer inside that row's features β CatBoost's contribution is Ilaya's rule, that a row may be described using everything the data knew before it and nothing the data learned after, which is not a clever optimisation but a refusal to look at tomorrow's newspaper while reporting today's news.
Follow me for the next article in the Boosting: The Complete Guide series!
If the court at Vashti made target leakage click, drop a heart!
Questions? Ask in the comments β I read and respond to every one.
Have you ever shipped a target-encoded feature without checking the fold boundaries? I have, and the model was the best I had ever built right up until the day it met data it hadn't already been told the answer to. βοΈ
The thing that unsettles me about leakage is that it never feels like cheating while you are doing it. Every step is defensible β use the data you have, summarise the category, feed the model context. The failure is assembled entirely from reasonable decisions, which is why code review does not catch it and a suspiciously good training curve does. Learn to distrust your best results specifically because they are your best results.