Claude3.5 Sonnet and GPT-4o to the test against a complex inheritance dispute involving 19th-century housing cooperative statutes. The goal wasn't just to "write a letter," but to perform a precise cross-reference of contradictory legal provisions.
The Technical Challenge: Dirty Data #
The input was a nightmare: a "PDF" that was actually a zip of images with no text layer. To make this a real-world AI workflow, I had to preprocess the data:
-
Unzip the image archive.
-
Rasterize and run OCR via a German-specific model.
-
Feed the resulting imperfect text (where the section symbol § was often misread as $) into the LLM.
In 2019, this would have been a dead end. Today, the "noise" in the OCR didn't stop the reasoning, provided the model had a large enough context window to hold the entire set of bylaws and the correspondence.
Model Performance: Claude 3.5 Sonnet vs. GPT-4o #
I ran the same set of documents through both models to see which could actually spot the legal error. The error was subtle: the cooperative was applying rules for inter vivos transfers (between living people), which require board approval, rather than mortis causa transfers (inheritance), which happen automatically by operation of law.
Reasoning Accuracy: Claude 3.5 Sonnet identified the specific paragraph conflict immediately. It mapped the "transfer" clause against the "inheritance" clause and flagged the contradiction. GPT-4o tended to summarize the bylaws generally but missed the specific nuance of the "automatic" nature of the inheritance.Citation Precision: Claude provided a reference document mapping every legal provision in play. It cited four specific statutes and three bylaw sections with near-perfect accuracy despite the OCR errors.Hallucination Rate: GPT-4o occasionally "invented" a standard probate procedure that didn't exist in the provided bylaws, whereas Claude stuck strictly to the provided context.Output Utility: Claude generated a formal letter that didn't sound like "AI prose." It used the specific terminology of the bylaws to corner the administrator into a logical dead end.
Practical Implementation: The Prompt Strategy #
To get this result, you cannot use a simple "summarize this" prompt. You need a multi-step prompt engineering approach to force the model into a "legal auditor" persona.
You are a senior legal analyst specializing in cooperative law. Your goal is to find contradictions between the provided Bylaws and the Correspondence.
1. Extract all mentions of "share transfer" and "inheritance/death" from the Bylaws.
2. Compare these provisions against the claims made in the Correspondence.
3. Identify if the Correspondence is applying a provision that is superseded by another.
4. Provide a list of citations (Section/Paragraph) to support your finding.
If the OCR text is corrupted (e.g., '$' instead of '§'), infer the meaning from context but flag the correction.
The Verdict #
For high-stakes reasoning where a single paragraph determines the outcome, Claude 3.5 Sonnet is currently superior to GPT-4o. The ability to handle "garbage" OCR text and perform a side-by-side comparison of conflicting rules makes it a legitimate tool for personal legal auditing.
The result of this specific deep dive was a two-page letter that proved the cooperative had applied the wrong paragraph. I didn't need a law degree; I just needed a model that could actually read.
Next Jixp: Lisp DSL for Jax Neural Nets →