{"slug": "how-we-cut-target-leakage-from-92-to-0-1-in-court-outcome-prediction-and-built-a", "title": "How We Cut Target Leakage from 92% to 0.1% in Court Outcome Prediction (and Built a Triple-A MCP Server)", "summary": "A developer built a court outcome prediction system that reduced target leakage from 92% to 0.1% across 609,715 Dutch court rulings by stripping outcome-announcing sentences before training. The system uses a traceable LightGBM model instead of an LLM, achieving 78.2% out-of-fold accuracy against a 43.7% majority baseline, and is exposed as a zero-dependency MCP server with three tools for AI assistants and agents.", "body_md": "When building AI for court outcome prediction, there is a massive hidden trap that invalidates most benchmarks: target leakage.\n\nIn Dutch court rulings, roughly 92% of raw texts contain the actual outcome verbatim (dictum or conclusion sentences like \"the court dismisses the claim\"). If you feed raw text to a model, it does not learn legal logic. It simply learns to read the answer back to you.\n\nHere is how we solved this across 609,715 cases, built a traceable LightGBM model, and exposed it as a zero-dependency open-core MCP server.\n\n# \n  \n  \n  1. The Pre-Training Cut: 92% to 0.1% Leakage\n\nBefore training any classifier, we implemented a strict sanitization step:\n\n- The dictum, summary lines, and outcome-announcing phrases are stripped from the text.\n- We continuously measure residual outcome markers.\n- Result: Leakage dropped from 92% to 0.1% (around 1 in 1,000 texts).\n\nOnly on this sanitized dataset did we train.\n\n# \n  \n  \n  2. Why LightGBM Instead of an LLM\n\nWe intentionally picked LightGBM over deep neural networks or fine-tuned LLMs:\n\n- Fast and cheap: Sub-10ms inference without GPUs.\n- Traceable: Clear tree structures and feature importance.\n- Deterministic calibration: If confidence drops below 55%, the model does not guess. It returns \"insufficient certainty\".\n\n# \n  \n  \n  3. Benchmark on 609,715 Cases (Out-of-Fold)\n\nEvaluated through 5-fold cross-validation, strictly measured out-of-fold:\n\n- Overall Accuracy: 78.2% (against a 43.7% majority baseline)\n- Macro-F1: 77.1%\n- Per-Class F1:\n  - Dismissed: 0.827\n  - Partly granted: 0.726\n  - Granted: 0.761\n\nDomain Breakdown:\n\n- Criminal Law (n=105,151): 82.6% accuracy, 0.804 macro-F1 (strongest performance)\n- Administrative Law (n=316,273): 81.0% accuracy, 0.685 macro-F1 (high accuracy, but government victory is the majority class)\n- Civil Law (n=188,177): 71.0% accuracy, 0.656 macro-F1 (most complex due to factual nuances)\n\n# \n  \n  \n  4. Model Context Protocol (MCP) Interface\n\nTo make this accessible to AI assistants (Claude, Cursor, autonomous agents), we wrapped the pipeline into an MCP server.\n\n- Zero external dependencies: Single Python file using only standard library (sys, json, urllib).\n- Audited on Glama: Triple-A rating (5/5 on coherence and completeness).\n- 3 Tools:\n  1. rechtspraak_cijfers (keyless): Benchmark statistics and baseline metrics.\n  2. lekkage_check (keyless): Paste any legal text to test for outcome leakage before and after the cut.\n  3. voorspel_uitkomst (key required): Outcome risk classification.\n\n# \n  \n  \n  Links\n\nDisclaimer: Not legal advice. Built as an open, verifiable yardstick for legal tech developers and agents.", "url": "https://wpnews.pro/news/how-we-cut-target-leakage-from-92-to-0-1-in-court-outcome-prediction-and-built-a", "canonical_source": "https://dev.to/rechtssysteem/how-we-cut-target-leakage-from-92-to-01-in-court-outcome-prediction-and-built-a-triple-a-mcp-478g", "published_at": "2026-09-11 00:39:12+00:00", "updated_at": "2026-09-11 01:22:13.791686+00:00", "lang": "en", "topics": ["machine-learning", "ai-tools", "ai-agents", "natural-language-processing", "developer-tools"], "entities": ["LightGBM", "Model Context Protocol", "Claude", "Cursor", "Glama"], "alternates": {"html": "https://wpnews.pro/news/how-we-cut-target-leakage-from-92-to-0-1-in-court-outcome-prediction-and-built-a", "markdown": "https://wpnews.pro/news/how-we-cut-target-leakage-from-92-to-0-1-in-court-outcome-prediction-and-built-a.md", "text": "https://wpnews.pro/news/how-we-cut-target-leakage-from-92-to-0-1-in-court-outcome-prediction-and-built-a.txt", "jsonld": "https://wpnews.pro/news/how-we-cut-target-leakage-from-92-to-0-1-in-court-outcome-prediction-and-built-a.jsonld"}}