Claude 3.5 Sonnet vs. GPT-4o: Legal Reasoning Benchmarks

Claude 3.5 Sonnet outperformed GPT-4o in a legal reasoning benchmark involving a complex inheritance dispute with 19th-century housing cooperative statutes, according to a test by a developer. Claude identified the specific paragraph conflict immediately, cited four statutes and three bylaw sections with near-perfect accuracy despite OCR errors, and generated a formal letter using specific bylaw terminology, while GPT-4o missed the nuance and occasionally invented nonexistent probate procedures. The test used a multi-step prompt engineering approach to force the model into a 'legal auditor' persona.

Claude 3.5 Sonnet vs. GPT-4o: Legal Reasoning Benchmarks Claude /en/tags/claude/ 3.5 Sonnet and GPT-4o to the test against a complex inheritance dispute involving 19th-century housing cooperative statutes. The goal wasn't just to "write a letter," but to perform a precise cross-reference of contradictory legal provisions. The Technical Challenge: Dirty Data The input was a nightmare: a "PDF" that was actually a zip of images with no text layer. To make this a real-world AI workflow, I had to preprocess the data: 1. Unzip the image archive. 2. Rasterize and run OCR via a German-specific model. 3. Feed the resulting imperfect text where the section symbol § was often misread as $ into the LLM. In 2019, this would have been a dead end. Today, the "noise" in the OCR didn't stop the reasoning, provided the model had a large enough context window to hold the entire set of bylaws and the correspondence. Model Performance: Claude 3.5 Sonnet vs. GPT-4o I ran the same set of documents through both models to see which could actually spot the legal error. The error was subtle: the cooperative was applying rules for inter vivos transfers between living people , which require board approval, rather than mortis causa transfers inheritance , which happen automatically by operation of law. Reasoning Accuracy: Claude 3.5 Sonnet identified the specific paragraph conflict immediately. It mapped the "transfer" clause against the "inheritance" clause and flagged the contradiction. GPT-4o tended to summarize the bylaws generally but missed the specific nuance of the "automatic" nature of the inheritance. Citation Precision: Claude provided a reference document mapping every legal provision in play. It cited four specific statutes and three bylaw sections with near-perfect accuracy despite the OCR errors. Hallucination Rate: GPT-4o occasionally "invented" a standard probate procedure that didn't exist in the provided bylaws, whereas Claude stuck strictly to the provided context. Output Utility: Claude generated a formal letter that didn't sound like "AI prose." It used the specific terminology of the bylaws to corner the administrator into a logical dead end. Practical Implementation: The Prompt Strategy To get this result, you cannot use a simple "summarize this" prompt. You need a multi-step prompt engineering approach to force the model into a "legal auditor" persona. System Prompt You are a senior legal analyst specializing in cooperative law. Your goal is to find contradictions between the provided Bylaws and the Correspondence. Task 1. Extract all mentions of "share transfer" and "inheritance/death" from the Bylaws. 2. Compare these provisions against the claims made in the Correspondence. 3. Identify if the Correspondence is applying a provision that is superseded by another. 4. Provide a list of citations Section/Paragraph to support your finding. Constraint If the OCR text is corrupted e.g., '$' instead of '§' , infer the meaning from context but flag the correction. The Verdict For high-stakes reasoning where a single paragraph determines the outcome, Claude 3.5 Sonnet is currently superior to GPT-4o. The ability to handle "garbage" OCR text and perform a side-by-side comparison of conflicting rules makes it a legitimate tool for personal legal auditing. The result of this specific deep dive was a two-page letter that proved the cooperative had applied the wrong paragraph. I didn't need a law degree; I just needed a model that could actually read. Next Jixp: Lisp DSL for Jax Neural Nets → /en/threads/2937/