A custom-grammar bot can be reliable right up until someone types a perfectly reasonable sentence that the grammar author did not anticipate.
The uncomfortable choice is usually framed as either:
That is the wrong boundary. You can keep command execution deterministic while using failed conversations to improve the grammar—and optionally use AI to propose interpretations that a human must confirm.
The key is to treat every misunderstanding as a potential regression test, not as an invitation to make the parser more permissive on the spot.
This tutorial builds that workflow for a small JavaScript bot that understands commands such as:
feed Luna 20g
remind me to feed Miso at 19:30
We will add support for natural variations without allowing an AI-generated guess to execute a command.
Our bot will produce one of three outcomes:
// Parsed successfully and passed domain validation.
{ status: "accepted", command: { ... }, grammarVersion: "2026-08-02" }
// The grammar did not recognize the utterance.
{ status: "no_match", grammarVersion: "2026-08-02" }
// It matched syntactically but violated a domain rule.
{ status: "invalid", reason: "amount_out_of_range", grammarVersion: "2026-08-02" }
Only accepted
commands are eligible for execution. A model suggestion, support reply, or partially parsed command is never equivalent to accepted
.
That distinction matters because the user’s real concern is not whether AI can produce a plausible interpretation. It often can. The concern is whether a plausible but wrong interpretation can silently become an action.
Create the project:
mkdir grammar-repair-loop
cd grammar-repair-loop
npm init -y
npm install peggy zod
npm install --save-dev vitest
mkdir -p src test data
Update package.json
:
{
"type": "module",
"scripts": {
"build:grammar": "peggy src/commands.peggy -o src/generated-parser.js --format es",
"test": "npm run build:grammar && vitest run"
}
}
Create src/commands.peggy
:
{
function feed(cat, amount) {
return { intent: "feed", cat, amountGrams: Number(amount) };
}
function reminder(cat, time) {
return { intent: "remind_feed", cat, time };
}
}
Start
= _ Please? command:(Reminder / Feed) Punctuation? _ { return command; }
Feed
= "feed"i __ cat:Name __ amount:Integer _ Unit {
return feed(cat, amount);
}
Reminder
= "remind"i __ "me"i __ "to"i __ "feed"i __ cat:Name __ "at"i __ time:Time {
return reminder(cat, time);
}
Please
= "please"i __
Name
= QuotedName / BareName
QuotedName
= '"' chars:[^"]+ '"' { return chars.join(""); }
BareName
= first:[A-Za-z] rest:[A-Za-z0-9_-]* {
return first + rest.join("");
}
Integer
= digits:[0-9]+ { return digits.join(""); }
Time
= hour:[0-9][0-9]? ":" minute:[0-9][0-9] {
return hour.join("") + ":" + minute.join("");
}
Unit
= "grams"i / "gram"i / "g"i
Punctuation
= [.!?]
_ = [ \t\n\r]*
__ = [ \t\n\r]+
The grammar handles polite prefixes, optional punctuation, and quoted multiword names. It still does not decide whether 99:99
is a valid time or whether feeding 5,000 grams is sensible. Those are domain decisions, so they belong outside the grammar.
Create src/parse.js
:
import { parse } from "./generated-parser.js";
export const GRAMMAR_VERSION = "2026-08-02";
function validTime(value) {
const match = /^(\d{1,2}):(\d{2})$/.exec(value);
if (!match) return false;
const hour = Number(match[1]);
const minute = Number(match[2]);
return hour >= 0 && hour <= 23 && minute >= 0 && minute <= 59;
}
function validate(command) {
if (command.intent === "feed") {
if (command.amountGrams < 1 || command.amountGrams > 200) {
return { ok: false, reason: "amount_out_of_range" };
}
}
if (command.intent === "remind_feed" && !validTime(command.time)) {
return { ok: false, reason: "invalid_time" };
}
return { ok: true };
}
export function parseCommand(rawUtterance) {
try {
const command = parse(rawUtterance);
const validation = validate(command);
if (!validation.ok) {
return {
status: "invalid",
reason: validation.reason,
grammarVersion: GRAMMAR_VERSION
};
}
return {
status: "accepted",
command,
grammarVersion: GRAMMAR_VERSION
};
} catch {
return {
status: "no_match",
grammarVersion: GRAMMAR_VERSION
};
}
}
Do not send parser exception text directly to users. Detailed parser errors are useful to developers, but messages such as “expected Unit
at offset 17” are rarely useful support responses and may expose grammar internals.
A chat transcript explains what happened once. A fixture prevents it from happening again.
Create data/grammar-cases.json
:
[
{
"id": "basic-feed",
"utterance": "feed Luna 20g",
"expected": {
"intent": "feed",
"cat": "Luna",
"amountGrams": 20
}
},
{
"id": "polite-feed",
"utterance": "Please feed Luna 20 grams.",
"expected": {
"intent": "feed",
"cat": "Luna",
"amountGrams": 20
}
},
{
"id": "quoted-name-reminder",
"utterance": "remind me to feed \"Sir Pounce\" at 07:30",
"expected": {
"intent": "remind_feed",
"cat": "Sir Pounce",
"time": "07:30"
}
}
]
Then add test/grammar.test.js
:
import { describe, expect, it } from "vitest";
import cases from "../data/grammar-cases.json" with { type: "json" };
import { parseCommand } from "../src/parse.js";
describe("command grammar", () => {
for (const fixture of cases) {
it(fixture.id, () => {
const result = parseCommand(fixture.utterance);
expect(result.status).toBe("accepted");
expect(result.command).toEqual(fixture.expected);
});
}
it("rejects an excessive feeding amount", () => {
expect(parseCommand("feed Luna 5000g")).toMatchObject({
status: "invalid",
reason: "amount_out_of_range"
});
});
it("does not reinterpret unrelated conversation", () => {
expect(parseCommand("Luna already ate today")).toMatchObject({
status: "no_match"
});
});
});
Run the corpus:
npm test
When a new wording fails, add it to the corpus before changing the grammar. The desired sequence is:
That last step is more important than simply reaching a green test suite.
A useful repair record needs the original outcome and the confirmed intent. It should not assume that the first developer—or a model—guessed correctly.
const repairCase = {
id: crypto.randomUUID(),
rawUtterance: "could you give Luna twenty grams?",
grammarVersion: "2026-08-02",
parserOutcome: "no_match",
// Added only after explicit confirmation.
confirmedCommand: null,
source: "in_app_support",
consentToRetainExample: false,
status: "needs_clarification",
createdAt: new Date().toISOString()
};
A practical state sequence is:
needs_clarification
-> intent_confirmed
-> fixture_added
-> grammar_changed
-> verified
-> released
Do not automatically retain every failed utterance. Commands can contain names, schedules, account details, or text pasted into the wrong box. Ask for permission before preserving an utterance as a long-lived test fixture, or replace identifying values with representative placeholders.
For example, a confirmed report containing a real pet name could become:
{
"utterance": "could you give ExampleCat twenty grams?",
"expected": {
"intent": "feed",
"cat": "ExampleCat",
"amountGrams": 20
}
}
A language model can be useful after no_match
because it can propose likely structured interpretations of paraphrases such as “could you give Luna twenty grams?” It can also draft a candidate fixture or group similar failures for maintainers.
That demonstrated capability is not the same as reliable command execution.
A model may:
Represent model output as a proposal and validate its shape:
import { z } from "zod";
const candidateSchema = z.discriminatedUnion("intent", [
z.object({
intent: z.literal("feed"),
cat: z.string().min(1),
amountGrams: z.number().int().min(1).max(200)
}),
z.object({
intent: z.literal("remind_feed"),
cat: z.string().min(1),
time: z.string().regex(/^\d{1,2}:\d{2}$/)
})
]);
export function reviewModelCandidate(rawCandidate, knownCats) {
const parsed = candidateSchema.safeParse(rawCandidate);
if (!parsed.success) return { usable: false, reason: "invalid_shape" };
if (!knownCats.includes(parsed.data.cat)) {
return { usable: false, reason: "unknown_entity" };
}
return {
usable: true,
proposal: parsed.data,
requiresUserConfirmation: true
};
}
Even a usable
proposal is not executable. It can power a clarification such as:
Did you mean “feed Luna 20 grams”?
The user’s confirmation can submit a fresh deterministic command. It should not retroactively turn the model response into authorization.
A simple decision framework is:
| Situation | AI role | Human control |
|---|---|---|
| Drafting a regression fixture | Suggest wording and expected structure | Maintainer approves the fixture |
| Grouping similar failures | Recommend clusters | Maintainer decides whether one grammar rule covers them |
| Read-only lookup | Suggest an interpretation | User confirms if ambiguity affects the result |
| Scheduling or changing state | Clarification only | Confirmed command passes deterministic validation |
| Billing, security, deletion, or irreversible action | Do not infer authorization | Use explicit, authenticated controls |
The useful boundary is based on consequence, not on how fluent the model sounds.
Grammar changes are code changes, but their most important diff may not appear in the grammar file. A reordered PEG alternative can change how existing inputs are interpreted while all new tests pass.
Before release, compare the previous and proposed parsers across the approved corpus. Flag these transitions for human review:
no_match -> accepted expected expansion
accepted -> no_match likely regression
accepted -> accepted inspect if the command AST changed
invalid -> accepted verify the domain rule intentionally changed
For every accepted -> accepted
transition, compare the complete command object rather than only the intent name. Changing amountGrams
from 20
to 200
is a semantic regression even though the intent remains feed
.
Keep the grammar version in support records so maintainers can reproduce behavior from the time of the report. A fixture without its parser version can become impossible to diagnose after several releases.
A user writes “feed Luna at seven.” A maintainer assumes seven grams, while the user meant seven o’clock.
Do not add a fixture until the intended meaning is confirmed. Some utterances should remain ambiguous and trigger clarification permanently.
PEG parsers use ordered choices. Adding a permissive rule above a specific rule can silently change which branch wins.
Include fixtures for overlapping forms, and review semantic diffs whenever alternatives are reordered.
A button labeled Continue may conceal what will happen. Display the proposed command in domain language:
Feed Luna 20 grams now
Require a separate confirmation event, then submit the confirmed structured command through the normal validation path.
A corpus containing only valid commands rewards increasingly permissive parsing. Keep negative fixtures for unrelated conversation, malformed times, excessive values, and unsupported actions.
If the report omits the grammar version, original parser outcome, or exact approved wording, a maintainer may test against different behavior. Capture those fields automatically while allowing the user to edit or decline the retained example.
The workflow does not depend on a particular support tool. A GitHub issue template, an email alias, or an in-app chat can all collect a failed utterance and ask the user to confirm the intended command.
For a small product, direct founder-led chat can shorten clarification because the person changing the grammar can ask a precise follow-up. The trade-off is that chat is conversational transport, not your regression corpus or release record. Confirmed cases still need to move into version-controlled fixtures.
One implementation option is Knocket, which provides a shareable contact page, an embeddable web live-chat widget, a mobile WebView SDK, and a unified inbox. Its website widget can be installed with a script tag without building a custom backend. Visitors do not need an account to begin a chat, and messages can be routed to Telegram; a quoted Telegram reply can be delivered back to the website visitor.
That can provide the clarification surface, but the parser result, consent decision, confirmed intent, fixture, and release verification should remain in systems you can test and audit.
Before shipping a grammar repair, verify:
A grammar bug is not fully fixed when one sentence starts parsing. It is fixed when the intended meaning is documented, the old behavior is reproducible, unrelated inputs remain safe, and future changes cannot silently undo the repair.
Disclosure: I work on Knocket, so treat it as one implementation example rather than a neutral recommendation.