{"slug": "valid-json-wrong-answer-semantic-checks-for-llm-output-in-n8n", "title": "Valid JSON, Wrong Answer: Semantic Checks for LLM Output in n8n", "summary": "A developer running LLM workflows in n8n described how a support-triage pipeline passed strict JSON schema validation for weeks while silently misclassifying a double-charge complaint as a low-priority feature request, because schema checks verify shape but not meaning. The proposed fix layers deterministic semantic checks on top of schema validation: cross-field consistency rules, business-derived plausibility ranges, and regex or keyword re-derivation of disputed fields, with disagreements routed to human review rather than retried. The developer also maintains a monthly regression set of about twenty tricky inputs, tracking semantic mismatches rather than schema errors as the metric for deciding whether to keep a prompt or model change.", "body_md": "*This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is from real templates I sell and support — the mistakes are mine, the fixes are tested.*\n\nA support-triage workflow of mine had every guard I thought it needed. The LLM returned strict JSON. A schema node validated the shape: `category` was a string, `priority` was one of `low`, `medium`, `high`, `amount` was a number. Every field present, every type correct, zero validation errors for weeks.\n\nThen a customer wrote in about a double charge, and the workflow filed it as `category: \"feature_request\"`, `priority: \"low\"`. The JSON was perfect. The meaning was nonsense. Nobody noticed for days, because the schema node was green and green means done — the same blind spot I wrote about with alerts that never fire. A schema checks that the answer has the right shape. It says nothing about whether the answer is true.\n\nThis is the failure mode that survives every shape-level fix: the model is confident, fluent, and wrong in a way that still parses. If your only check is \"did it return valid JSON,\" you are testing the envelope, not the letter.\n\nThe fix is a second layer of checks that do not come from the model at all. They come from the input itself, from deterministic rules, and from the relationships between fields.\n\nFirst, cross-field consistency. If `category` is `refund` but the message contains no money, no order number, and no word like \"charged\" or \"refund,\" that is a contradiction, not a judgment call. If `priority` is `high` but `amount` is zero and no urgency words appear, something is off. These checks are cheap — a keyword list, a regex, a comparison between two fields — and they catch exactly the class of error a schema cannot see: two individually valid values that cannot both be true.\n\nSecond, plausibility ranges from the business, not the prompt. An extracted `amount` of $10,000 on a $29 product is not a rounding error; it is a misread decimal or a hallucinated zero. A `date` in the past for a \"schedule for tomorrow\" request is a misread. Hard-code the ranges your business actually has — product prices, valid regions, working hours — and reject anything outside them. The model does not know your price list. Your workflow should.\n\nThird, re-derive, don't re-ask. When a check fails, the instinct is to retry the same prompt and hope for a better answer. Usually you get the same confident wrong answer back, at full token cost — the retry trap I wrote about with fallback chains. Instead, recompute the disputed field deterministically where you can: extract the amount with a regex from the original text, look up the category from a keyword map, compare. If the deterministic answer and the model answer disagree, you have learned something real: this input is ambiguous. Ambiguous inputs do not get retried. They go to review, with both answers attached, so a human decides once instead of a model guessing three times.\n\nA semantic-check failure that lands in review without context is just another queue nobody watches. Each flagged item carries the original input excerpt, the model's answer, the rule that failed, and the deterministic re-derivation. That is enough for a human to decide in under a minute, and it is also a dataset: every resolved disagreement is a case you can add to a test set.\n\nThat test set is the real payoff. Once a month, I run new prompt or model changes against twenty known tricky inputs — sarcasm, mixed requests (\"refund this and also upgrade that\"), numbers written as words, dates without years — and count semantic mismatches, not schema errors. Schema errors are already zero; they tell me nothing. Semantic mismatches are the metric that actually moves. If a model change drops mismatches from five to two, keep it. If it raises them, roll back, even if the new model is faster and cheaper. Speed on wrong answers is not an improvement.\n\nThe rule I now build by: shape validation proves the model followed instructions. Semantic checks prove it understood the input. You need both, and only one of them can be written as a schema. The other has to be written as rules about your business — which is, inconveniently, the part no prompt can do for you.\n\n*I'm building [AI Automation Lab](https://payhip.com/haopengailab) — production-ready n8n workflow templates. The free [AI Hook Generator](https://payhip.com/b/ulER8) shows how I structure real workflows; my open-source templates live at [github.com/zhp910318/n8n-ai-automation-lab](https://github.com/zhp910318/n8n-ai-automation-lab).*", "url": "https://wpnews.pro/news/valid-json-wrong-answer-semantic-checks-for-llm-output-in-n8n", "canonical_source": "https://dev.to/haopengai/valid-json-wrong-answer-semantic-checks-for-llm-output-in-n8n-3cg5", "published_at": "2026-10-23 21:33:57+00:00", "updated_at": "2026-10-23 21:39:54.875456+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "mlops"], "entities": ["n8n"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/valid-json-wrong-answer-semantic-checks-for-llm-output-in-n8n", "markdown": "https://wpnews.pro/news/valid-json-wrong-answer-semantic-checks-for-llm-output-in-n8n.md", "text": "https://wpnews.pro/news/valid-json-wrong-answer-semantic-checks-for-llm-output-in-n8n.txt", "jsonld": "https://wpnews.pro/news/valid-json-wrong-answer-semantic-checks-for-llm-output-in-n8n.jsonld"}}