Turn Chatbot Misunderstandings Into Grammar Regression Tests

A developer built a JavaScript bot that turns chatbot misunderstandings into grammar regression tests, using a custom grammar parser with domain validation to keep command execution deterministic while optionally using AI to propose interpretations that require human confirmation. The workflow, implemented with Peggy and Zod, ensures only accepted commands are executed, preventing AI-generated guesses from silently becoming actions.

A custom-grammar bot can be reliable right up until someone types a perfectly reasonable sentence that the grammar author did not anticipate. The uncomfortable choice is usually framed as either: That is the wrong boundary. You can keep command execution deterministic while using failed conversations to improve the grammar—and optionally use AI to propose interpretations that a human must confirm. The key is to treat every misunderstanding as a potential regression test, not as an invitation to make the parser more permissive on the spot. This tutorial builds that workflow for a small JavaScript bot that understands commands such as: feed Luna 20g remind me to feed Miso at 19:30 We will add support for natural variations without allowing an AI-generated guess to execute a command. Our bot will produce one of three outcomes: // Parsed successfully and passed domain validation. { status: "accepted", command: { ... }, grammarVersion: "2026-08-02" } // The grammar did not recognize the utterance. { status: "no match", grammarVersion: "2026-08-02" } // It matched syntactically but violated a domain rule. { status: "invalid", reason: "amount out of range", grammarVersion: "2026-08-02" } Only accepted commands are eligible for execution. A model suggestion, support reply, or partially parsed command is never equivalent to accepted . That distinction matters because the user’s real concern is not whether AI can produce a plausible interpretation. It often can. The concern is whether a plausible but wrong interpretation can silently become an action. Create the project: mkdir grammar-repair-loop cd grammar-repair-loop npm init -y npm install peggy zod npm install --save-dev vitest mkdir -p src test data Update package.json : { "type": "module", "scripts": { "build:grammar": "peggy src/commands.peggy -o src/generated-parser.js --format es", "test": "npm run build:grammar && vitest run" } } Create src/commands.peggy : { function feed cat, amount { return { intent: "feed", cat, amountGrams: Number amount }; } function reminder cat, time { return { intent: "remind feed", cat, time }; } } Start = Please? command: Reminder / Feed Punctuation? { return command; } Feed = "feed"i cat:Name amount:Integer Unit { return feed cat, amount ; } Reminder = "remind"i "me"i "to"i "feed"i cat:Name "at"i time:Time { return reminder cat, time ; } Please = "please"i Name = QuotedName / BareName QuotedName = '"' chars: ^" + '"' { return chars.join "" ; } BareName = first: A-Za-z rest: A-Za-z0-9 - { return first + rest.join "" ; } Integer = digits: 0-9 + { return digits.join "" ; } Time = hour: 0-9 0-9 ? ":" minute: 0-9 0-9 { return hour.join "" + ":" + minute.join "" ; } Unit = "grams"i / "gram"i / "g"i Punctuation = . ? = \t\n\r = \t\n\r + The grammar handles polite prefixes, optional punctuation, and quoted multiword names. It still does not decide whether 99:99 is a valid time or whether feeding 5,000 grams is sensible. Those are domain decisions, so they belong outside the grammar. Create src/parse.js : js import { parse } from "./generated-parser.js"; export const GRAMMAR VERSION = "2026-08-02"; function validTime value { const match = /^ \d{1,2} : \d{2} $/.exec value ; if match return false; const hour = Number match 1 ; const minute = Number match 2 ; return hour = 0 && hour <= 23 && minute = 0 && minute <= 59; } function validate command { if command.intent === "feed" { if command.amountGrams < 1 || command.amountGrams 200 { return { ok: false, reason: "amount out of range" }; } } if command.intent === "remind feed" && validTime command.time { return { ok: false, reason: "invalid time" }; } return { ok: true }; } export function parseCommand rawUtterance { try { const command = parse rawUtterance ; const validation = validate command ; if validation.ok { return { status: "invalid", reason: validation.reason, grammarVersion: GRAMMAR VERSION }; } return { status: "accepted", command, grammarVersion: GRAMMAR VERSION }; } catch { return { status: "no match", grammarVersion: GRAMMAR VERSION }; } } Do not send parser exception text directly to users. Detailed parser errors are useful to developers, but messages such as “expected Unit at offset 17” are rarely useful support responses and may expose grammar internals. A chat transcript explains what happened once. A fixture prevents it from happening again. Create data/grammar-cases.json : { "id": "basic-feed", "utterance": "feed Luna 20g", "expected": { "intent": "feed", "cat": "Luna", "amountGrams": 20 } }, { "id": "polite-feed", "utterance": "Please feed Luna 20 grams.", "expected": { "intent": "feed", "cat": "Luna", "amountGrams": 20 } }, { "id": "quoted-name-reminder", "utterance": "remind me to feed \"Sir Pounce\" at 07:30", "expected": { "intent": "remind feed", "cat": "Sir Pounce", "time": "07:30" } } Then add test/grammar.test.js : python import { describe, expect, it } from "vitest"; import cases from "../data/grammar-cases.json" with { type: "json" }; import { parseCommand } from "../src/parse.js"; describe "command grammar", = { for const fixture of cases { it fixture.id, = { const result = parseCommand fixture.utterance ; expect result.status .toBe "accepted" ; expect result.command .toEqual fixture.expected ; } ; } it "rejects an excessive feeding amount", = { expect parseCommand "feed Luna 5000g" .toMatchObject { status: "invalid", reason: "amount out of range" } ; } ; it "does not reinterpret unrelated conversation", = { expect parseCommand "Luna already ate today" .toMatchObject { status: "no match" } ; } ; } ; Run the corpus: npm test When a new wording fails, add it to the corpus before changing the grammar. The desired sequence is: That last step is more important than simply reaching a green test suite. A useful repair record needs the original outcome and the confirmed intent. It should not assume that the first developer—or a model—guessed correctly. js const repairCase = { id: crypto.randomUUID , rawUtterance: "could you give Luna twenty grams?", grammarVersion: "2026-08-02", parserOutcome: "no match", // Added only after explicit confirmation. confirmedCommand: null, source: "in app support", consentToRetainExample: false, status: "needs clarification", createdAt: new Date .toISOString }; A practical state sequence is: php needs clarification - intent confirmed - fixture added - grammar changed - verified - released Do not automatically retain every failed utterance. Commands can contain names, schedules, account details, or text pasted into the wrong box. Ask for permission before preserving an utterance as a long-lived test fixture, or replace identifying values with representative placeholders. For example, a confirmed report containing a real pet name could become: { "utterance": "could you give ExampleCat twenty grams?", "expected": { "intent": "feed", "cat": "ExampleCat", "amountGrams": 20 } } A language model can be useful after no match because it can propose likely structured interpretations of paraphrases such as “could you give Luna twenty grams?” It can also draft a candidate fixture or group similar failures for maintainers. That demonstrated capability is not the same as reliable command execution. A model may: Represent model output as a proposal and validate its shape: js import { z } from "zod"; const candidateSchema = z.discriminatedUnion "intent", z.object { intent: z.literal "feed" , cat: z.string .min 1 , amountGrams: z.number .int .min 1 .max 200 } , z.object { intent: z.literal "remind feed" , cat: z.string .min 1 , time: z.string .regex /^\d{1,2}:\d{2}$/ } ; export function reviewModelCandidate rawCandidate, knownCats { const parsed = candidateSchema.safeParse rawCandidate ; if parsed.success return { usable: false, reason: "invalid shape" }; if knownCats.includes parsed.data.cat { return { usable: false, reason: "unknown entity" }; } return { usable: true, proposal: parsed.data, requiresUserConfirmation: true }; } Even a usable proposal is not executable. It can power a clarification such as: Did you mean “feed Luna 20 grams”? The user’s confirmation can submit a fresh deterministic command. It should not retroactively turn the model response into authorization. A simple decision framework is: | Situation | AI role | Human control | |---|---|---| | Drafting a regression fixture | Suggest wording and expected structure | Maintainer approves the fixture | | Grouping similar failures | Recommend clusters | Maintainer decides whether one grammar rule covers them | | Read-only lookup | Suggest an interpretation | User confirms if ambiguity affects the result | | Scheduling or changing state | Clarification only | Confirmed command passes deterministic validation | | Billing, security, deletion, or irreversible action | Do not infer authorization | Use explicit, authenticated controls | The useful boundary is based on consequence, not on how fluent the model sounds. Grammar changes are code changes, but their most important diff may not appear in the grammar file. A reordered PEG alternative can change how existing inputs are interpreted while all new tests pass. Before release, compare the previous and proposed parsers across the approved corpus. Flag these transitions for human review: php no match - accepted expected expansion accepted - no match likely regression accepted - accepted inspect if the command AST changed invalid - accepted verify the domain rule intentionally changed For every accepted - accepted transition, compare the complete command object rather than only the intent name. Changing amountGrams from 20 to 200 is a semantic regression even though the intent remains feed . Keep the grammar version in support records so maintainers can reproduce behavior from the time of the report. A fixture without its parser version can become impossible to diagnose after several releases. A user writes “feed Luna at seven.” A maintainer assumes seven grams, while the user meant seven o’clock. Do not add a fixture until the intended meaning is confirmed. Some utterances should remain ambiguous and trigger clarification permanently. PEG parsers use ordered choices. Adding a permissive rule above a specific rule can silently change which branch wins. Include fixtures for overlapping forms, and review semantic diffs whenever alternatives are reordered. A button labeled Continue may conceal what will happen. Display the proposed command in domain language: Feed Luna 20 grams now Require a separate confirmation event, then submit the confirmed structured command through the normal validation path. A corpus containing only valid commands rewards increasingly permissive parsing. Keep negative fixtures for unrelated conversation, malformed times, excessive values, and unsupported actions. If the report omits the grammar version, original parser outcome, or exact approved wording, a maintainer may test against different behavior. Capture those fields automatically while allowing the user to edit or decline the retained example. The workflow does not depend on a particular support tool. A GitHub issue template, an email alias, or an in-app chat can all collect a failed utterance and ask the user to confirm the intended command. For a small product, direct founder-led chat can shorten clarification because the person changing the grammar can ask a precise follow-up. The trade-off is that chat is conversational transport, not your regression corpus or release record. Confirmed cases still need to move into version-controlled fixtures. One implementation option is Knocket https://knocket.trtc.io/ , which provides a shareable contact page, an embeddable web live-chat widget, a mobile WebView SDK, and a unified inbox. Its website widget can be installed with a script tag without building a custom backend. Visitors do not need an account to begin a chat, and messages can be routed to Telegram; a quoted Telegram reply can be delivered back to the website visitor. That can provide the clarification surface, but the parser result, consent decision, confirmed intent, fixture, and release verification should remain in systems you can test and audit. Before shipping a grammar repair, verify: A grammar bug is not fully fixed when one sentence starts parsing. It is fixed when the intended meaning is documented, the old behavior is reproducible, unrelated inputs remain safe, and future changes cannot silently undo the repair. Disclosure: I work on Knocket, so treat it as one implementation example rather than a neutral recommendation.