{"slug": "catboost-the-interpreter-who-refused-to-peek-at-tomorrow-s-newspaper", "title": "CatBoost: The Interpreter Who Refused to Peek at Tomorrow's Newspaper", "summary": "A developer's blog post explains how CatBoost's ordered target statistics prevent target leakage in categorical feature encoding, a common issue in tabular machine learning. The post illustrates the problem with an analogy of court interpreters who, given the case file including the verdict, appear skilled on old cases but are merely leaking future information. CatBoost's method ensures each row only sees data from preceding rows, avoiding the leak that can make training loss misleadingly low and test loss five times worse.", "body_md": "The One-Line Summary:Replacing a category with the average target for that category is the most common feature-engineering trick in tabular machine learning and it quietly hands the model the answer — measured here, it turned a column containingliterally no signalinto a training loss of 0.2466 and a test loss of 2.7975, five times worse than deleting the column; CatBoost's ordered target statistics fix the leak by letting each row see only rows that came before it.\n\nThe court employed interpreters, because testimony arrived in six languages and the judges read one. An interpreter's job was to render each witness's words faithfully, and for two centuries the court considered this a solved problem.\n\nThen someone noticed that the interpreters were extraordinarily good at old cases and merely average at new ones.\n\nEvery interpreter was handed the complete case file before beginning. This was considered basic professionalism — how could you translate testimony about a boundary dispute without knowing it was a boundary dispute?\n\nThe file included the verdict.\n\n```\nWHY THE OLD INTERPRETERS LOOKED SO GOOD\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nWitness says a word that could mean\n\"he took it\" or \"he was given it\".\n\nInterpreter has read the file. The file says\nGUILTY.\n\nShe writes \"he took it.\"\n\nShe is not lying. She is not even conscious of\nchoosing. The ambiguity resolved itself the\nmoment she knew how it ended.\n\nOn closed cases her renderings are uncanny.\nOn open cases she is ordinary, and nobody can\nwork out why.\n```\n\nThe interpreters were not corrupt. They were *contaminated*, which is worse, because contamination leaves no one to blame and nothing obvious to fix.\n\nA clerk named Ilaya was asked to audit the interpreters and did something nobody had tried: she took a hundred old cases, stripped the verdicts, and had them re-translated by the same people.\n\n```\nTHE SAME INTERPRETERS, VERDICTS HIDDEN\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nwith the file (as always)   near-perfect renderings\nwithout the verdict          ordinary renderings\n                             indistinguishable from\n                             a first-year apprentice\n\nThe skill everyone had been admiring for two\ncenturies was not skill. It was the verdict,\ntravelling backwards into the testimony.\n```\n\nThe court's instinct was to ban case files entirely. Ilaya argued against it: context genuinely helps, and an interpreter working blind is worse than one working informed. The problem was never the file. It was *which parts of the file*.\n\n\"Let her read everything that was written before the words she is translating. Nothing that was written after. The rule is not ignorance — it is chronology.\"\n\n```\nILAYA'S RULE\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nTranslating testimony from the 3rd of the month?\n\n  You may read: everything filed on the 1st, 2nd.\n  You may not read: the 4th onward. Ever.\n                    Especially the verdict.\n\nEach document is translated using only the\ncourt's knowledge AS IT STOOD at that moment.\n\nSlower. Occasionally the interpreter has almost\nno context and must simply admit it.\nThose renderings are honest, and they hold up.\n```\n\nThe first interpreters trained this way looked *worse* on the court's historical records. They were the first ones who did not get worse when the case was live.\n\nHard pivot.\n\nYou have a categorical column with many levels — city, merchant, user ID, product SKU. One-hot encoding it produces thousands of columns. The popular alternative is **target encoding**: replace each level with the mean of the target for that level.\n\nLook at the index set. It includes\nj=i\n. **Row\ni\n's own label is in the numerator of row\ni\n's feature.** If a level appears once, its encoding *is* its label, exactly. If it appears three times, the encoding is two-thirds someone else's answer and one-third your own.\n\nThat is the verdict travelling backwards into the testimony.\n\nCatBoost computes **ordered target statistics**. Fix a random permutation\nσ\nof the rows. For row\ni\n, use only rows that precede it:\n\nwhere p is a prior (the global mean) and a its weight. Row i is excluded by construction — it hasn't happened yet. The prior does the work when history is thin.\n\nThe same principle is applied to the *gradients* during boosting, which is where the name \"ordered boosting\" comes from, but the target-statistic version is where the damage usually is.\n\nThe cleanest possible test: build a categorical column that carries **no information whatsoever**, and see whether target encoding can conjure signal out of it. 6,000 rows, 2,000 levels — roughly three rows per level, which is exactly the regime where people reach for target encoding.\n\n``` python\nimport numpy as np, pandas as pd\nfrom sklearn.model_selection import train_test_split, KFold\nfrom sklearn.metrics import log_loss\nfrom sklearn.ensemble import HistGradientBoostingClassifier\n\nrng = np.random.default_rng(0)\nn, K = 6000, 2000\ncat = rng.integers(0, K, n)              # pure noise: 2,000 levels\nx1 = rng.normal(size=n)\ny = (0.9 * x1 + rng.normal(0, 1.0, n) > 0).astype(int)   # y ignores cat\n\nX = pd.DataFrame({\"cat\": cat, \"x1\": x1})\nXtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)\nctr, cte = Xtr[\"cat\"].values, Xte[\"cat\"].values\ngm = ytr.mean()\n\ndef score(A, B):\n    m = HistGradientBoostingClassifier(max_iter=200, random_state=0).fit(A, ytr)\n    return (log_loss(ytr, m.predict_proba(A)[:, 1]),\n            log_loss(yte, m.predict_proba(B)[:, 1]))\n\nfull_mean = pd.Series(ytr).groupby(ctr).mean()\nB = Xte.copy(); B[\"cat\"] = pd.Series(cte).map(full_mean).fillna(gm).values\n\nA = Xtr.copy(); A[\"cat\"] = pd.Series(ctr).map(full_mean).fillna(gm).values\ntr, te = score(A, B)\nprint(f\"  naive target encoding   train {tr:.4f}   test {te:.4f}   gap {te-tr:+.4f}\")\n\nA2 = Xtr.copy(); enc = np.full(len(Xtr), np.nan); ys = pd.Series(ytr)\nfor itr, iva in KFold(5, shuffle=True, random_state=0).split(Xtr):\n    mm = ys.iloc[itr].groupby(ctr[itr]).mean()\n    enc[iva] = pd.Series(ctr[iva]).map(mm).fillna(ys.iloc[itr].mean()).values\nA2[\"cat\"] = enc\ntr, te = score(A2, B)\nprint(f\"  out-of-fold encoding    train {tr:.4f}   test {te:.4f}   gap {te-tr:+.4f}\")\n\ntr, te = score(Xtr[[\"x1\"]], Xte[[\"x1\"]])\nprint(f\"  drop the column         train {tr:.4f}   test {te:.4f}   gap {te-tr:+.4f}\")\nnaive target encoding   train 0.2466   test 2.7975   gap +2.5508\n  out-of-fold encoding    train 0.4436   test 0.5823   gap +0.1388\n  drop the column         train 0.4964   test 0.5464   gap +0.0500\n```\n\nSit with the first row.\n\nThe column is **random integers**. It has no relationship to the target — I generated `y`\n\nfrom `x1`\n\nalone. And naive target encoding produced a **training loss of 0.2466**, better than the honest model can achieve, and a **test loss of 2.7975** — roughly five times worse than simply deleting the column.\n\nA model that has memorised the training set and learned nothing looks, during training, exactly like a breakthrough.\n\nIlaya's rule is about thirty lines. Walk the rows in a random permutation, encode each one from the running totals so far, *then* add it to the totals.\n\n``` python\nimport numpy as np\n\ndef ordered_target_stats(cat, y, prior, a=1.0, seed=0):\n    \"\"\"Each row is encoded using only rows earlier in the permutation.\"\"\"\n    rng = np.random.default_rng(seed)\n    perm = rng.permutation(len(cat))\n    enc = np.empty(len(cat))\n    csum, ccnt = {}, {}\n    for pos in perm:\n        c = cat[pos]\n        s, k = csum.get(c, 0.0), ccnt.get(c, 0)\n        enc[pos] = (s + a * prior) / (k + a)   # history only — self excluded\n        csum[c] = s + y[pos]                   # now add self, for later rows\n        ccnt[c] = k + 1\n    return enc\n```\n\nThe two lines after the assignment are the whole idea. The row contributes to *everyone after it* and never to itself.\n\nRun all four encodings on the same data, both when the column is noise and when it genuinely carries signal, three seeds each:\n\n```\n=== NO signal in the column (test logloss, 3 seeds) ===\n  seed   naive    oof      ordered   drop\n     0   2.7975   0.5823   0.5812   0.5464\n     1   3.2548   0.5780   0.6050   0.5545\n     2   2.9780   0.5988   0.6014   0.5702\n  MEAN   3.0101   0.5863   0.5959   0.5570\n\n=== REAL signal in the column (test logloss, 3 seeds) ===\n  seed   naive    oof      ordered   drop\n     0   2.2485   0.5707   0.6737   0.6398\n     1   2.5480   0.5564   0.6239   0.6139\n     2   2.0538   0.5601   0.6670   0.6518\n  MEAN   2.2834   0.5624   0.6549   0.6352\n```\n\nFour readings, and the third one surprised me.\n\n**Naive encoding is a catastrophe either way** — 3.0101 and 2.2834 against roughly 0.56 for every honest method. It does not matter whether the column has signal. The leak dominates.\n\n**When the column carries real signal, encoding it pays.** Out-of-fold scored 0.5624 against 0.6352 for dropping the column. That is the case for target encoding, and it is a real one.\n\n**My ordered implementation lost to plain out-of-fold** — 0.6549 against 0.5624. I expected the opposite, and I am reporting it because it is what the code printed.\n\n**Dropping a signal-free column beats every encoding of it.** 0.5570 against 0.5863 out-of-fold. No scheme recovers information that was never there; encoding only adds noise with a plausible face.\n\nA result you did not expect is worth ten minutes before you publish it. The obvious suspect is variance: early rows in the permutation have almost no history, so their encodings are mostly prior. CatBoost's actual answer to this is to average several permutations — so I tried that.\n\n``` python\nimport numpy as np, pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import log_loss\nfrom sklearn.ensemble import HistGradientBoostingClassifier\n\ndef ordered_target_stats(cat, y, prior, a=1.0, seed=0):\n    rng = np.random.default_rng(seed)\n    perm = rng.permutation(len(cat))\n    enc = np.empty(len(cat)); csum, ccnt = {}, {}\n    for pos in perm:\n        c = cat[pos]; s, k = csum.get(c, 0.0), ccnt.get(c, 0)\n        enc[pos] = (s + a * prior) / (k + a)\n        csum[c] = s + y[pos]; ccnt[c] = k + 1\n    return enc\n\nrng = np.random.default_rng(0)\nn, K = 6000, 2000\ncat = rng.integers(0, K, n)\neff = rng.normal(0, 1.5, K)                     # this time the column MATTERS\nx1 = rng.normal(size=n)\ny = (eff[cat] + 0.9 * x1 + rng.normal(0, 1.0, n) > 0).astype(int)\nX = pd.DataFrame({\"cat\": cat, \"x1\": x1})\nXtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)\nctr, cte = Xtr[\"cat\"].values, Xte[\"cat\"].values\ngm = ytr.mean()\nfm = pd.Series(ytr).groupby(ctr).mean()\nB = Xte.copy(); B[\"cat\"] = pd.Series(cte).map(fm).fillna(gm).values\n\nprint(\"  permutations   test logloss\")\nfor P in (1, 2, 4, 8, 16):\n    encs = np.mean([ordered_target_stats(ctr, ytr, gm, seed=s)\n                    for s in range(P)], axis=0)\n    A = Xtr.copy(); A[\"cat\"] = encs\n    m = HistGradientBoostingClassifier(max_iter=200, random_state=0).fit(A, ytr)\n    print(f\"  {P:>12}   {log_loss(yte, m.predict_proba(B)[:, 1]):.4f}\")\n\n# and the diagnostic that explains the result\nperm = np.random.default_rng(0).permutation(len(ctr))\nseen, hist = {}, []\nfor pos in perm:\n    c = ctr[pos]; hist.append(seen.get(c, 0)); seen[c] = seen.get(c, 0) + 1\nhist = np.array(hist)\nprint(f\"\\n  rows with ZERO prior observations of their own level: \"\n      f\"{(hist==0).sum():,} of {len(hist):,} ({(hist==0).mean()*100:.1f}%)\")\nprint(f\"  first 10% of the permutation: \"\n      f\"{(hist[:len(hist)//10]==0).mean()*100:.1f}% have no history\")\nprint(f\"  last  10% of the permutation: \"\n      f\"{(hist[-len(hist)//10:]==0).mean()*100:.1f}% have no history\")\npermutations   test logloss\n             1   0.6737\n             2   0.7020\n             4   0.7110\n             8   0.6810\n            16   0.6854\n```\n\nAveraging did not help. So the problem is not variance, and I went looking for what it actually is:\n\n```\n  rows with ZERO prior observations of their own level: 1,747 of 4,200 (41.6%)\n  first 10% of the permutation: 92.1% have no history\n  last  10% of the permutation: 13.1% have no history\n```\n\n**41.6% of training rows never see a single earlier example of their own category.** At 2,000 levels across 4,200 training rows, most levels appear two or three times, so for a large fraction of rows there is simply nothing legitimate to learn from. Their encoding is the global prior — a constant. Averaging permutations cannot manufacture history that does not exist.\n\nOut-of-fold encoding wins here because it is less strict: a row in fold 3 gets the mean over folds 1, 2, 4 and 5 — around 80% of the data rather than \"whatever happened to precede me\". Both exclude the row's own label, which is the part that matters. Ordered TS excludes more, and at this cardinality it excludes too much.\n\n**One caveat I want to be explicit about:** this is my thirty-line reimplementation of the principle, not CatBoost. Real CatBoost recomputes target statistics inside the boosting loop with a fresh permutation per tree, builds combinations of categorical features, and applies the same ordering idea to gradients. Do not read this table as \"CatBoost loses to out-of-fold encoding\" — read it as \"the ordering principle, applied naively as a preprocessing step, runs out of history at extreme cardinality.\"\n\n```\nTARGET ENCODING: A DECISION PROCEDURE\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nDoes the column plausibly carry signal?\n  |\n  +- No  -> DROP IT. Measured best in every\n  |         no-signal run. Encoding a useless\n  |         column only invents noise.\n  |\n  +- Yes / unsure\n       |\n       +- Fewer than ~50 levels?\n       |    -> one-hot. Boring, safe, competitive.\n       |\n       +- Many levels, several rows each?\n       |    -> out-of-fold target encoding.\n       |       Best measured result here.\n       |\n       +- Many levels, 1-3 rows each?\n            -> the column is nearly an ID.\n               Expect little. Consider dropping,\n               or hashing to fewer buckets.\n\nNEVER: mean over all training rows including self.\n       train 0.2466 / test 2.7975 on a column\n       that contained nothing.\n```\n\nThe tell that you have leaked is always the same shape: **training loss far better than anything the problem should allow, and a validation gap that widens as you add capacity.** A gap of +2.55 log loss is not overfitting. Overfitting is +0.05. That is contamination.\n\n| CatBoost | LightGBM | XGBoost | |\n|---|---|---|---|\n| Categorical handling | ordered target statistics, built in | native partition or one-hot | you encode it yourself |\n| Leak protection | by construction | none — your job | none — your job |\n| Tree shape | oblivious (same split per level) | leaf-wise | depth-wise |\n| Defaults on small data | conservative | dangerous (`num_leaves=31` ) |\nsafe |\n| Best fit | many high-cardinality categoricals | large numeric tabular | general tabular |\n\nReach for CatBoost when your table is mostly categorical with real cardinality — retail SKUs, merchant IDs, geography — and you would otherwise be hand-rolling encodings and getting them subtly wrong. Reach elsewhere when your features are numeric, where its advantage largely disappears.\n\n```\nCATBOOST / TARGET ENCODING: CHEAT SHEET\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nTHE LEAK:\n  x_i = mean(y_j for j where c_j == c_i)\n  includes j == i.  Row i's own label is inside\n  row i's feature. One row per level => the\n  encoding IS the label.\n\nTHE FIX (ordered target statistics):\n  fix a permutation; for row i use only rows\n  BEFORE it:\n    x_i = (sum of earlier y + a*prior)\n          / (count of earlier + a)\n  self excluded by construction.\n\nMEASURED (column with NO signal at all):\n  naive        train 0.2466   test 2.7975\n  out-of-fold  train 0.4436   test 0.5823\n  drop it      train 0.4964   test 0.5464   <- best\n\nMEASURED (column WITH signal):\n  naive 2.2834 | oof 0.5624 | ordered 0.6549\n  | drop 0.6352      oof wins\n\nWHY ORDERED CAN LOSE:\n  41.6% of rows had NO earlier row of their own\n  level (2,000 levels / 4,200 rows). Averaging\n  16 permutations did not help — the history\n  isn't thin, it's absent.\n\nTHE TELL YOU LEAKED:\n  train loss impossibly good + gap that GROWS\n  with capacity. Overfitting is +0.05.\n  Leakage was +2.55.\n```\n\n**Naive target encoding leaks the label into the feature.** The row's own\ny\nsits in the numerator of its own\nx\n.\n\n**It manufactures signal from nothing.** On a column of random integers: train 0.2466, test 2.7975 — five times worse than deleting the column.\n\n**Ordered target statistics fix it by chronology, not ignorance.** Each row sees only rows earlier in a random permutation; it is excluded from its own encoding by construction.\n\n**Out-of-fold encoding was the best real method measured** — 0.5624 against 0.6352 for dropping a signal-bearing column, and it beat my ordered implementation.\n\n**Dropping a signal-free column beats every encoding of it** (0.5570 vs 0.5863). Encoding cannot recover information that was never present.\n\n**Ordering can exclude too much.** 41.6% of rows had no earlier example of their own level, and averaging 16 permutations did not help — the fix for absent history is not more permutations.\n\n**Leakage and overfitting look different.** Overfitting widened the gap by 0.05. Leakage widened it by 2.55. If your training metric looks impossible, it is.\n\n**Check your own reimplementation before blaming the library.** My thirty-line ordered TS is not CatBoost, and I said so rather than reporting a benchmark I had not actually run.\n\n**Target encoding is the most widely used trick in tabular machine learning and its textbook form quietly puts each row's own answer inside that row's features — CatBoost's contribution is Ilaya's rule, that a row may be described using everything the data knew before it and nothing the data learned after, which is not a clever optimisation but a refusal to look at tomorrow's newspaper while reporting today's news.**\n\nFollow me for the next article in the **Boosting: The Complete Guide** series!\n\nIf the court at Vashti made target leakage click, drop a heart!\n\n**Questions?** Ask in the comments — I read and respond to every one.\n\n**Have you ever shipped a target-encoded feature without checking the fold boundaries?** I have, and the model was the best I had ever built right up until the day it met data it hadn't already been told the answer to. ⚖️\n\n*The thing that unsettles me about leakage is that it never feels like cheating while you are doing it. Every step is defensible — use the data you have, summarise the category, feed the model context. The failure is assembled entirely from reasonable decisions, which is why code review does not catch it and a suspiciously good training curve does. Learn to distrust your best results specifically because they are your best results.*", "url": "https://wpnews.pro/news/catboost-the-interpreter-who-refused-to-peek-at-tomorrow-s-newspaper", "canonical_source": "https://dev.to/sachin_krrajput/catboost-the-interpreter-who-refused-to-peek-at-tomorrows-newspaper-2nag", "published_at": "2026-08-12 07:19:38+00:00", "updated_at": "2026-08-12 07:46:52.154858+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "mlops"], "entities": ["CatBoost"], "alternates": {"html": "https://wpnews.pro/news/catboost-the-interpreter-who-refused-to-peek-at-tomorrow-s-newspaper", "markdown": "https://wpnews.pro/news/catboost-the-interpreter-who-refused-to-peek-at-tomorrow-s-newspaper.md", "text": "https://wpnews.pro/news/catboost-the-interpreter-who-refused-to-peek-at-tomorrow-s-newspaper.txt", "jsonld": "https://wpnews.pro/news/catboost-the-interpreter-who-refused-to-peek-at-tomorrow-s-newspaper.jsonld"}}