# Luc Julia claims LLMs only hit 64% reliability and I want to see > Source: > Published: 2026-08-27 16:23:29+00:00 # Luc Julia claims LLMs only hit 64% reliability and I want to see Reliability in this context isn't just about whether a model "works," but whether it follows complex, multi-step logic without hallucinating or drifting from the constraints. When we talk about a 64% reliability rate, we are likely looking at high-reasoning tasks—things like code generation, mathematical proofs, or complex instruction following—rather than simple chat interactions. ## The anatomy of a reliability benchmark To replicate or verify such a claim, you can't just ask a model "Are you reliable?" You need a rigorous testing pipeline. A real-world benchmark for LLM reliability typically requires three specific components: 1. **Deterministic Ground Truth:** You need a dataset where there is only one objectively correct answer (like a unit test for code or a specific mathematical result). 2. **Constraint Adherence Scoring:** It's not enough to get the answer right; the model must follow the formatting. If the prompt asks for JSON and the model returns a conversational sentence containing the answer, that is a failure in a production AI workflow. 3. **The Reasoning Gap:** Measuring the "Chain of Thought" (CoT) accuracy. If the model reaches the right answer through flawed logic, is it actually reliable? In a strict benchmark, the answer is no. ## Comparing the heavyweights on reliability Based on my own hands-on testing and various recent evaluations, the "64% problem" varies wildly depending on which model you are deploying. Currently feels like the gold standard for instruction following. In my experience, when tasks involve strict schema adherence (JSON/YAML), Claude stays much closer to the 80-90% range than the 64% cited by Julia.[Claude](/en/tags/claude/)3.5 Sonnet:**GPT-4o:** Extremely capable but occasionally prone to "laziness" or skipping steps in long-form reasoning, which can tank reliability scores in complex, multi-turn agentic workflows.Shows incredible strength in coding and logic-heavy benchmarks, often rivaling the top-tier US models, though their reliability in nuanced, natural language instruction following is still being mapped out by the community.[DeepSeek](/en/tags/deepseek/)-V3/R1: ## How to move past the 64% ceiling If you are building an LLM agent, you cannot accept 64% reliability. You don't solve this by waiting for a "smarter" model; you solve it through architecture. A practical tutorial for increasing reliability involves moving away from single-shot prompting toward an iterative loop. Instead of:`Prompt -> LLM -> Output` You should implement:`Prompt -> LLM -> Critic/Validator LLM -> Self-Correction Loop -> Final Output` By using a secondary, smaller model (like a specialized Llama-3 or a distilled version) to act as a "validator" that checks the primary model's output against the original constraints, you can push that 64% reliability up to a much more usable 95%+. The cost is higher latency and higher token usage, but that is the tax we pay for moving from a playground experiment to a real-world deployment. [Next The traditional 3-step voice AI pipeline is fundamentally broken →](/en/threads/7907/) [a library of Claude prompt techniques](https://tanyan888.com/), with plenty of directly applicable cases.