When building AI for court outcome prediction, there is a massive hidden trap that invalidates most benchmarks: target leakage.

In Dutch court rulings, roughly 92% of raw texts contain the actual outcome verbatim (dictum or conclusion sentences like "the court dismisses the claim"). If you feed raw text to a model, it does not learn legal logic. It simply learns to read the answer back to you.

Here is how we solved this across 609,715 cases, built a traceable LightGBM model, and exposed it as a zero-dependency open-core MCP server.

#

1. The Pre-Training Cut: 92% to 0.1% Leakage

Before training any classifier, we implemented a strict sanitization step:

• The dictum, summary lines, and outcome-announcing phrases are stripped from the text.

• We continuously measure residual outcome markers.

• Result: Leakage dropped from 92% to 0.1% (around 1 in 1,000 texts). Only on this sanitized dataset did we train.

#

2. Why LightGBM Instead of an LLM

We intentionally picked LightGBM over deep neural networks or fine-tuned LLMs:

• Fast and cheap: Sub-10ms inference without GPUs.
• Traceable: Clear tree structures and feature importance.
• Deterministic calibration: If confidence drops below 55%, the model does not guess. It returns "insufficient certainty".

#

  3. Benchmark on 609,715 Cases (Out-of-Fold)

Evaluated through 5-fold cross-validation, strictly measured out-of-fold:

- Overall Accuracy: 78.2% (against a 43.7% majority baseline)
- Macro-F1: 77.1%
- Per-Class F1:
  - Dismissed: 0.827
  - Partly granted: 0.726
  - Granted: 0.761

Domain Breakdown:

- Criminal Law (n=105,151): 82.6% accuracy, 0.804 macro-F1 (strongest performance)
- Administrative Law (n=316,273): 81.0% accuracy, 0.685 macro-F1 (high accuracy, but government victory is the majority class)
- Civil Law (n=188,177): 71.0% accuracy, 0.656 macro-F1 (most complex due to factual nuances)

#

4. Model Context Protocol (MCP) Interface

To make this accessible to AI assistants (Claude, Cursor, autonomous agents), we wrapped the pipeline into an MCP server.

• Zero external dependencies: Single Python file using only standard library (sys, json, urllib).

- Audited on Glama: Triple-A rating (5/5 on coherence and completeness).
- 3 Tools:

1. rechtspraak_cijfers (keyless): Benchmark statistics and baseline metrics.
2. lekkage_check (keyless): Paste any legal text to test for outcome leakage before and after the cut.
3. voorspel_uitkomst (key required): Outcome risk classification.

#

Links

Disclaimer: Not legal advice. Built as an open, verifiable yardstick for legal tech developers and agents.