{"slug": "common-llm-failure-modes", "title": "Common LLM failure modes", "summary": "Anthropic's Claude Opus 5 and Claude Fable 5 exhibit distinct failure modes, including Opus 5's constant praise, imprecise commentary, and a tendency to end statements with negations, while Fable 5 incorrectly uses discrete values for continuous variables in Weights & Biases sweeps. The observations, documented by a researcher, aim to track behavior changes across model versions.", "body_md": "I use the various language models a *lot* for my research, and this means I end\nup seeing quite a lot of their failure modes. I’m going to try track of them\nhere, as an archive for my future self to answer “did this behaviour start with\nOpus 4.7 or 4.8”?\n\n# Claude Fable 5\n\n## W&B sweeps using discrete values for continuous variables\n\nFor some reason, Fable (and I think Opus as well, but I can’t remember) seem to like using discrete values in W&B sweeps, when the variable is continuous and so should rather be a continuous variable:\n\nIn this snippet, β is continuous but Fable sets it to be one of the following\nvalues: `[0, 0.005, 0.01, 0.02, 0.05, 0.1]`\n\n.\n\nA β sweep it is — better experiment design anyway. Let me read the sweep machinery first (config shape, how pods/deadlines are computed for long runs), since 5M-step runs are ~3h each and the pod budgeting has to fit.\n\n[…]\n\nReady on the branch(head`d57715e`\n\n):\n\n`ci/sweep_ppo.yaml`\n\nnow sweepsβ ∈ {0, 0.005, 0.01, 0.02, 0.05, 0.1} × seeds {1,2,3}= 18 runs, each 5M steps from`h76h80yb`\n\n, metric`eval/thput`\n\n,`run_cap: 18`\n\n. β=0 is the control arm, 0.1 re-tests the known over-anchor at the longer horizon, and the log-spacing brackets the projected 0.2–0.3 nat sweet spot.\n\n# Claude Opus 5\n\n## Constant praise for the user’s ideas\n\nLess obvious than the blatant sycophancy of previous Claudes, but still there’s so many “you’re right” and “good call”\n\n`You're right, and that's a real error on my part. [...]`\n\n`Fair challenge. [...]`\n\n`Good call. [...]`\n\n`Clear. [...]`\n\n`You're right, and the thing I missed makes most of that spec redundant.`\n\n## Imprecise commentary\n\nOpus has a real tendency to say things without actually saying anything. These\ncomments communicate *nothing* but add a weird sense of anticipation that’s not\nhelpful.\n\n```\nGood call. Let me verify one more thing first, because it changes what's worth writing down.\n\nBash(uv run --with pandas --with pyarrow python -c \"…\")\n\nThat changes things — this is the most useful fact in the dataset and nothing in the repo records it.\n```\n\n`Fair challenge. Here's exactly what I computed, and it's weaker evidence than my phrasing implied.`\n\n`You're right, and that's a real error on my part.`\n\n`The one thing I can't do — your call to run:`\n\n## A fascination with problems that “bite”\n\nOpus often talks about problems as “biting” us in various ways\n\n`[...] and the specific carve-out that bit that behaviour before [...]`\n\n`## Five things that will bite us`\n\n## ’The X worth naming’ or ‘The Y worth stating clearly’ etc\n\nOpus 5 likes announcing & signposting its sentences in this way.\n\n`The trap worth naming:`\n\n## Ending with a negation\n\nOpus likes ending phrases with negations such as “and neither matter” or “and one is the exception”, e.g.:\n\n`[...] Let me check the one part that isn't trivial.`\n\n`Concretely: two places build a sv2.py command line, and neither would have carried --root.`\n\n`Clear. That makes the review a hard gate on vector creation, not a step someone can forget.`\n\n`Both questions land on the same correction, and it goes against what I recommended.`\n\n`Two different situations, and only one should be edited.`\n\n`Two different problems, and only one was actually a wrong file:`\n\n## ’rather than reciting from memory’\n\nSome posttraining process traumatised opus against ever just saying something, the facts always need a 30s tool call to figure out.\n\n`Let me read the actual definitions rather than paraphrase from memory.`", "url": "https://wpnews.pro/news/common-llm-failure-modes", "canonical_source": "https://boydkane.com/essays/common-llm-failure-modes", "published_at": "2026-08-26 13:47:25+00:00", "updated_at": "2026-08-26 14:14:20.782580+00:00", "lang": "en", "topics": ["large-language-models", "ai-research"], "entities": ["Anthropic", "Claude Opus 5", "Claude Fable 5", "Weights & Biases"], "alternates": {"html": "https://wpnews.pro/news/common-llm-failure-modes", "markdown": "https://wpnews.pro/news/common-llm-failure-modes.md", "text": "https://wpnews.pro/news/common-llm-failure-modes.txt", "jsonld": "https://wpnews.pro/news/common-llm-failure-modes.jsonld"}}