How We Cut Target Leakage from 92% to 0.1% in Court Outcome Prediction (and Built a Triple-A MCP Server)

A developer built a court outcome prediction system that reduced target leakage from 92% to 0.1% across 609,715 Dutch court rulings by stripping outcome-announcing sentences before training. The system uses a traceable LightGBM model instead of an LLM, achieving 78.2% out-of-fold accuracy against a 43.7% majority baseline, and is exposed as a zero-dependency MCP server with three tools for AI assistants and agents.

When building AI for court outcome prediction, there is a massive hidden trap that invalidates most benchmarks: target leakage. In Dutch court rulings, roughly 92% of raw texts contain the actual outcome verbatim dictum or conclusion sentences like "the court dismisses the claim" . If you feed raw text to a model, it does not learn legal logic. It simply learns to read the answer back to you. Here is how we solved this across 609,715 cases, built a traceable LightGBM model, and exposed it as a zero-dependency open-core MCP server. 1. The Pre-Training Cut: 92% to 0.1% Leakage Before training any classifier, we implemented a strict sanitization step: - The dictum, summary lines, and outcome-announcing phrases are stripped from the text. - We continuously measure residual outcome markers. - Result: Leakage dropped from 92% to 0.1% around 1 in 1,000 texts . Only on this sanitized dataset did we train. 2. Why LightGBM Instead of an LLM We intentionally picked LightGBM over deep neural networks or fine-tuned LLMs: - Fast and cheap: Sub-10ms inference without GPUs. - Traceable: Clear tree structures and feature importance. - Deterministic calibration: If confidence drops below 55%, the model does not guess. It returns "insufficient certainty". 3. Benchmark on 609,715 Cases Out-of-Fold Evaluated through 5-fold cross-validation, strictly measured out-of-fold: - Overall Accuracy: 78.2% against a 43.7% majority baseline - Macro-F1: 77.1% - Per-Class F1: - Dismissed: 0.827 - Partly granted: 0.726 - Granted: 0.761 Domain Breakdown: - Criminal Law n=105,151 : 82.6% accuracy, 0.804 macro-F1 strongest performance - Administrative Law n=316,273 : 81.0% accuracy, 0.685 macro-F1 high accuracy, but government victory is the majority class - Civil Law n=188,177 : 71.0% accuracy, 0.656 macro-F1 most complex due to factual nuances 4. Model Context Protocol MCP Interface To make this accessible to AI assistants Claude, Cursor, autonomous agents , we wrapped the pipeline into an MCP server. - Zero external dependencies: Single Python file using only standard library sys, json, urllib . - Audited on Glama: Triple-A rating 5/5 on coherence and completeness . - 3 Tools: 1. rechtspraak cijfers keyless : Benchmark statistics and baseline metrics. 2. lekkage check keyless : Paste any legal text to test for outcome leakage before and after the cut. 3. voorspel uitkomst key required : Outcome risk classification. Links Disclaimer: Not legal advice. Built as an open, verifiable yardstick for legal tech developers and agents.