{"slug": "turn-chatbot-misunderstandings-into-grammar-regression-tests", "title": "Turn Chatbot Misunderstandings Into Grammar Regression Tests", "summary": "A developer built a JavaScript bot that turns chatbot misunderstandings into grammar regression tests, using a custom grammar parser with domain validation to keep command execution deterministic while optionally using AI to propose interpretations that require human confirmation. The workflow, implemented with Peggy and Zod, ensures only accepted commands are executed, preventing AI-generated guesses from silently becoming actions.", "body_md": "A custom-grammar bot can be reliable right up until someone types a perfectly reasonable sentence that the grammar author did not anticipate.\n\nThe uncomfortable choice is usually framed as either:\n\nThat is the wrong boundary. You can keep command execution deterministic while using failed conversations to improve the grammar—and optionally use AI to propose interpretations that a human must confirm.\n\nThe key is to treat every misunderstanding as a potential regression test, not as an invitation to make the parser more permissive on the spot.\n\nThis tutorial builds that workflow for a small JavaScript bot that understands commands such as:\n\n```\nfeed Luna 20g\nremind me to feed Miso at 19:30\n```\n\nWe will add support for natural variations without allowing an AI-generated guess to execute a command.\n\nOur bot will produce one of three outcomes:\n\n```\n// Parsed successfully and passed domain validation.\n{ status: \"accepted\", command: { ... }, grammarVersion: \"2026-08-02\" }\n\n// The grammar did not recognize the utterance.\n{ status: \"no_match\", grammarVersion: \"2026-08-02\" }\n\n// It matched syntactically but violated a domain rule.\n{ status: \"invalid\", reason: \"amount_out_of_range\", grammarVersion: \"2026-08-02\" }\n```\n\nOnly `accepted`\n\ncommands are eligible for execution. A model suggestion, support reply, or partially parsed command is never equivalent to `accepted`\n\n.\n\nThat distinction matters because the user’s real concern is not whether AI can produce a plausible interpretation. It often can. The concern is whether a plausible but wrong interpretation can silently become an action.\n\nCreate the project:\n\n```\nmkdir grammar-repair-loop\ncd grammar-repair-loop\nnpm init -y\nnpm install peggy zod\nnpm install --save-dev vitest\nmkdir -p src test data\n```\n\nUpdate `package.json`\n\n:\n\n```\n{\n  \"type\": \"module\",\n  \"scripts\": {\n    \"build:grammar\": \"peggy src/commands.peggy -o src/generated-parser.js --format es\",\n    \"test\": \"npm run build:grammar && vitest run\"\n  }\n}\n```\n\nCreate `src/commands.peggy`\n\n:\n\n```\n{\n  function feed(cat, amount) {\n    return { intent: \"feed\", cat, amountGrams: Number(amount) };\n  }\n\n  function reminder(cat, time) {\n    return { intent: \"remind_feed\", cat, time };\n  }\n}\n\nStart\n  = _ Please? command:(Reminder / Feed) Punctuation? _ { return command; }\n\nFeed\n  = \"feed\"i __ cat:Name __ amount:Integer _ Unit {\n      return feed(cat, amount);\n    }\n\nReminder\n  = \"remind\"i __ \"me\"i __ \"to\"i __ \"feed\"i __ cat:Name __ \"at\"i __ time:Time {\n      return reminder(cat, time);\n    }\n\nPlease\n  = \"please\"i __\n\nName\n  = QuotedName / BareName\n\nQuotedName\n  = '\"' chars:[^\"]+ '\"' { return chars.join(\"\"); }\n\nBareName\n  = first:[A-Za-z] rest:[A-Za-z0-9_-]* {\n      return first + rest.join(\"\");\n    }\n\nInteger\n  = digits:[0-9]+ { return digits.join(\"\"); }\n\nTime\n  = hour:[0-9][0-9]? \":\" minute:[0-9][0-9] {\n      return hour.join(\"\") + \":\" + minute.join(\"\");\n    }\n\nUnit\n  = \"grams\"i / \"gram\"i / \"g\"i\n\nPunctuation\n  = [.!?]\n\n_  = [ \\t\\n\\r]*\n__ = [ \\t\\n\\r]+\n```\n\nThe grammar handles polite prefixes, optional punctuation, and quoted multiword names. It still does not decide whether `99:99`\n\nis a valid time or whether feeding 5,000 grams is sensible. Those are domain decisions, so they belong outside the grammar.\n\nCreate `src/parse.js`\n\n:\n\n``` js\nimport { parse } from \"./generated-parser.js\";\n\nexport const GRAMMAR_VERSION = \"2026-08-02\";\n\nfunction validTime(value) {\n  const match = /^(\\d{1,2}):(\\d{2})$/.exec(value);\n  if (!match) return false;\n\n  const hour = Number(match[1]);\n  const minute = Number(match[2]);\n  return hour >= 0 && hour <= 23 && minute >= 0 && minute <= 59;\n}\n\nfunction validate(command) {\n  if (command.intent === \"feed\") {\n    if (command.amountGrams < 1 || command.amountGrams > 200) {\n      return { ok: false, reason: \"amount_out_of_range\" };\n    }\n  }\n\n  if (command.intent === \"remind_feed\" && !validTime(command.time)) {\n    return { ok: false, reason: \"invalid_time\" };\n  }\n\n  return { ok: true };\n}\n\nexport function parseCommand(rawUtterance) {\n  try {\n    const command = parse(rawUtterance);\n    const validation = validate(command);\n\n    if (!validation.ok) {\n      return {\n        status: \"invalid\",\n        reason: validation.reason,\n        grammarVersion: GRAMMAR_VERSION\n      };\n    }\n\n    return {\n      status: \"accepted\",\n      command,\n      grammarVersion: GRAMMAR_VERSION\n    };\n  } catch {\n    return {\n      status: \"no_match\",\n      grammarVersion: GRAMMAR_VERSION\n    };\n  }\n}\n```\n\nDo not send parser exception text directly to users. Detailed parser errors are useful to developers, but messages such as “expected `Unit`\n\nat offset 17” are rarely useful support responses and may expose grammar internals.\n\nA chat transcript explains what happened once. A fixture prevents it from happening again.\n\nCreate `data/grammar-cases.json`\n\n:\n\n```\n[\n  {\n    \"id\": \"basic-feed\",\n    \"utterance\": \"feed Luna 20g\",\n    \"expected\": {\n      \"intent\": \"feed\",\n      \"cat\": \"Luna\",\n      \"amountGrams\": 20\n    }\n  },\n  {\n    \"id\": \"polite-feed\",\n    \"utterance\": \"Please feed Luna 20 grams.\",\n    \"expected\": {\n      \"intent\": \"feed\",\n      \"cat\": \"Luna\",\n      \"amountGrams\": 20\n    }\n  },\n  {\n    \"id\": \"quoted-name-reminder\",\n    \"utterance\": \"remind me to feed \\\"Sir Pounce\\\" at 07:30\",\n    \"expected\": {\n      \"intent\": \"remind_feed\",\n      \"cat\": \"Sir Pounce\",\n      \"time\": \"07:30\"\n    }\n  }\n]\n```\n\nThen add `test/grammar.test.js`\n\n:\n\n``` python\nimport { describe, expect, it } from \"vitest\";\nimport cases from \"../data/grammar-cases.json\" with { type: \"json\" };\nimport { parseCommand } from \"../src/parse.js\";\n\ndescribe(\"command grammar\", () => {\n  for (const fixture of cases) {\n    it(fixture.id, () => {\n      const result = parseCommand(fixture.utterance);\n\n      expect(result.status).toBe(\"accepted\");\n      expect(result.command).toEqual(fixture.expected);\n    });\n  }\n\n  it(\"rejects an excessive feeding amount\", () => {\n    expect(parseCommand(\"feed Luna 5000g\")).toMatchObject({\n      status: \"invalid\",\n      reason: \"amount_out_of_range\"\n    });\n  });\n\n  it(\"does not reinterpret unrelated conversation\", () => {\n    expect(parseCommand(\"Luna already ate today\")).toMatchObject({\n      status: \"no_match\"\n    });\n  });\n});\n```\n\nRun the corpus:\n\n```\nnpm test\n```\n\nWhen a new wording fails, add it to the corpus before changing the grammar. The desired sequence is:\n\nThat last step is more important than simply reaching a green test suite.\n\nA useful repair record needs the original outcome and the confirmed intent. It should not assume that the first developer—or a model—guessed correctly.\n\n``` js\nconst repairCase = {\n  id: crypto.randomUUID(),\n  rawUtterance: \"could you give Luna twenty grams?\",\n  grammarVersion: \"2026-08-02\",\n  parserOutcome: \"no_match\",\n\n  // Added only after explicit confirmation.\n  confirmedCommand: null,\n\n  source: \"in_app_support\",\n  consentToRetainExample: false,\n  status: \"needs_clarification\",\n  createdAt: new Date().toISOString()\n};\n```\n\nA practical state sequence is:\n\n``` php\nneeds_clarification\n  -> intent_confirmed\n  -> fixture_added\n  -> grammar_changed\n  -> verified\n  -> released\n```\n\nDo not automatically retain every failed utterance. Commands can contain names, schedules, account details, or text pasted into the wrong box. Ask for permission before preserving an utterance as a long-lived test fixture, or replace identifying values with representative placeholders.\n\nFor example, a confirmed report containing a real pet name could become:\n\n```\n{\n  \"utterance\": \"could you give ExampleCat twenty grams?\",\n  \"expected\": {\n    \"intent\": \"feed\",\n    \"cat\": \"ExampleCat\",\n    \"amountGrams\": 20\n  }\n}\n```\n\nA language model can be useful after `no_match`\n\nbecause it can propose likely structured interpretations of paraphrases such as “could you give Luna twenty grams?” It can also draft a candidate fixture or group similar failures for maintainers.\n\nThat demonstrated capability is not the same as reliable command execution.\n\nA model may:\n\nRepresent model output as a proposal and validate its shape:\n\n``` js\nimport { z } from \"zod\";\n\nconst candidateSchema = z.discriminatedUnion(\"intent\", [\n  z.object({\n    intent: z.literal(\"feed\"),\n    cat: z.string().min(1),\n    amountGrams: z.number().int().min(1).max(200)\n  }),\n  z.object({\n    intent: z.literal(\"remind_feed\"),\n    cat: z.string().min(1),\n    time: z.string().regex(/^\\d{1,2}:\\d{2}$/)\n  })\n]);\n\nexport function reviewModelCandidate(rawCandidate, knownCats) {\n  const parsed = candidateSchema.safeParse(rawCandidate);\n  if (!parsed.success) return { usable: false, reason: \"invalid_shape\" };\n\n  if (!knownCats.includes(parsed.data.cat)) {\n    return { usable: false, reason: \"unknown_entity\" };\n  }\n\n  return {\n    usable: true,\n    proposal: parsed.data,\n    requiresUserConfirmation: true\n  };\n}\n```\n\nEven a `usable`\n\nproposal is not executable. It can power a clarification such as:\n\nDid you mean “feed Luna 20 grams”?\n\nThe user’s confirmation can submit a fresh deterministic command. It should not retroactively turn the model response into authorization.\n\nA simple decision framework is:\n\n| Situation | AI role | Human control |\n|---|---|---|\n| Drafting a regression fixture | Suggest wording and expected structure | Maintainer approves the fixture |\n| Grouping similar failures | Recommend clusters | Maintainer decides whether one grammar rule covers them |\n| Read-only lookup | Suggest an interpretation | User confirms if ambiguity affects the result |\n| Scheduling or changing state | Clarification only | Confirmed command passes deterministic validation |\n| Billing, security, deletion, or irreversible action | Do not infer authorization | Use explicit, authenticated controls |\n\nThe useful boundary is based on consequence, not on how fluent the model sounds.\n\nGrammar changes are code changes, but their most important diff may not appear in the grammar file. A reordered PEG alternative can change how existing inputs are interpreted while all new tests pass.\n\nBefore release, compare the previous and proposed parsers across the approved corpus. Flag these transitions for human review:\n\n``` php\nno_match  -> accepted   expected expansion\naccepted -> no_match    likely regression\naccepted -> accepted    inspect if the command AST changed\ninvalid  -> accepted    verify the domain rule intentionally changed\n```\n\nFor every `accepted -> accepted`\n\ntransition, compare the complete command object rather than only the intent name. Changing `amountGrams`\n\nfrom `20`\n\nto `200`\n\nis a semantic regression even though the intent remains `feed`\n\n.\n\nKeep the grammar version in support records so maintainers can reproduce behavior from the time of the report. A fixture without its parser version can become impossible to diagnose after several releases.\n\nA user writes “feed Luna at seven.” A maintainer assumes seven grams, while the user meant seven o’clock.\n\nDo not add a fixture until the intended meaning is confirmed. Some utterances should remain ambiguous and trigger clarification permanently.\n\nPEG parsers use ordered choices. Adding a permissive rule above a specific rule can silently change which branch wins.\n\nInclude fixtures for overlapping forms, and review semantic diffs whenever alternatives are reordered.\n\nA button labeled **Continue** may conceal what will happen. Display the proposed command in domain language:\n\n```\nFeed Luna 20 grams now\n```\n\nRequire a separate confirmation event, then submit the confirmed structured command through the normal validation path.\n\nA corpus containing only valid commands rewards increasingly permissive parsing. Keep negative fixtures for unrelated conversation, malformed times, excessive values, and unsupported actions.\n\nIf the report omits the grammar version, original parser outcome, or exact approved wording, a maintainer may test against different behavior. Capture those fields automatically while allowing the user to edit or decline the retained example.\n\nThe workflow does not depend on a particular support tool. A GitHub issue template, an email alias, or an in-app chat can all collect a failed utterance and ask the user to confirm the intended command.\n\nFor a small product, direct founder-led chat can shorten clarification because the person changing the grammar can ask a precise follow-up. The trade-off is that chat is conversational transport, not your regression corpus or release record. Confirmed cases still need to move into version-controlled fixtures.\n\nOne implementation option is [Knocket](https://knocket.trtc.io/), which provides a shareable contact page, an embeddable web live-chat widget, a mobile WebView SDK, and a unified inbox. Its website widget can be installed with a script tag without building a custom backend. Visitors do not need an account to begin a chat, and messages can be routed to Telegram; a quoted Telegram reply can be delivered back to the website visitor.\n\nThat can provide the clarification surface, but the parser result, consent decision, confirmed intent, fixture, and release verification should remain in systems you can test and audit.\n\nBefore shipping a grammar repair, verify:\n\nA grammar bug is not fully fixed when one sentence starts parsing. It is fixed when the intended meaning is documented, the old behavior is reproducible, unrelated inputs remain safe, and future changes cannot silently undo the repair.\n\n**Disclosure:** I work on Knocket, so treat it as one implementation example rather than a neutral recommendation.", "url": "https://wpnews.pro/news/turn-chatbot-misunderstandings-into-grammar-regression-tests", "canonical_source": "https://dev.to/susiewang/turn-chatbot-misunderstandings-into-grammar-regression-tests-j5n", "published_at": "2026-08-02 04:11:25+00:00", "updated_at": "2026-08-02 04:37:55.495764+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence"], "entities": ["Peggy", "Zod", "Vitest"], "alternates": {"html": "https://wpnews.pro/news/turn-chatbot-misunderstandings-into-grammar-regression-tests", "markdown": "https://wpnews.pro/news/turn-chatbot-misunderstandings-into-grammar-regression-tests.md", "text": "https://wpnews.pro/news/turn-chatbot-misunderstandings-into-grammar-regression-tests.txt", "jsonld": "https://wpnews.pro/news/turn-chatbot-misunderstandings-into-grammar-regression-tests.jsonld"}}