When an AI workflow makes a mistake, it is tempting to change the model. But if a client email is almost archived or a routine receipt keeps reaching a person, the first question is more useful: which decision was wrong, and which part of the system should own it? We have been answering that question in the email triage system we use to run Kaizen. Its written rules, model decisions and checks in code can change separately.
The improvement loop is not a retraining program. It is a way to turn repeated decisions into inspectable rules while keeping high-cost mistakes in code and outbound actions behind a person's approval.
Give repeated decisions a written rule #
Our email triage checks written rules before asking a model. A rule can match an address, a domain or words in a subject, and the most specific match wins. People can read, change and turn off those rules. The model sees mail only when no rule covers it. In our published walkthrough of the system, we reported about 750 rules and about 90% of mail sorted by a rule in the preceding 30 days. Those are measurements from that write-up, not a promise about another inbox.
This makes a recurring receipt a rule-design question rather than a reason to ask the model the same thing again next month. It also lets a person inspect why a message was handled a certain way. A model's answer to one message is local; a rule changes what happens to future mail. We give those two decisions different evidence requirements.
Require evidence before the rule persists #
A confident verdict on one unfamiliar message can guide how that message is handled. It cannot, on its own, write a lasting rule. In our system, the model's verdict needs confidence of at least 0.85, and a second message from the same sender address must receive the same verdict before an automatic rule is written. The learned rule covers that exact address, not the whole company domain. A person must write any domain-wide rule.
The narrow scope matters because the same company can send automated receipts from one address and a human support reply from another. When a check in code overrules the model, that verdict is not used to learn a rule. Our triage explanation shows the sequence: the first message receives a verdict, the second agreeing one establishes a rule, and later mail from that address skips the model. That is improvement by accumulating evidence, not changing model weights.
A team can borrow the distinction even if its workflow has no email: ask how many observations justify a one-off action, and how many justify a rule that will affect future work. The second threshold should be harder to cross.
Put expensive mistakes in code #
Rules and models still make mistakes. For email, the mistake we most need to prevent is hiding a message someone should see. Checks in code run after either kind of verdict and can only move a message toward a person. A client contact on a message keeps it unread in the inbox; urgent words in the subject keep it visible and flagged. A message from someone we have written to is kept. None of these checks can archive mail.
The direction of the check matters as much as its condition. If the model is unsure, below 0.6 confidence, or the model call fails, the message is left alone. A client asking to schedule a meeting uses a separate path in code to create a work item and a phone alert. That path does not wait for a model classification. The public operating case study also shows why alerts open records instead of sending an untraceable notification.
When reviewing a failure, write down the damage it could have caused. Then choose the narrowest check that prevents it without asking a model to obey a stronger adjective in a prompt. A check can be tested and inspected independently of a later model change.
Leave sending to a person #
Better classification is not permission to take every action. Our agent drafts routine replies on review work items, but it has no email-sending tool. A person taps Send. The triage write-up describes the boundary, including the decision not to draft legal, pricing or personnel threads.
This is another kind of improvement: adjust the workflow around a model so that a doubtful judgment stays reversible. A wrong classification can be corrected. A message sent to someone outside the company cannot be unsent in the same way.
Start with a decision log, not a model project #
For one running workflow, take a handful of recent cases a person corrected. Separate the questions: Was the same decision being made repeatedly? Was there enough evidence to turn it into a narrow rule? Was the error too costly to leave to classification? Would the next step have gone outside the organization? Our email workflow routes those answers to rules, code checks and approval work items. Your workflow may need different boundaries, but the review should start with actual decisions and their consequences.
If you have a running AI workflow with a decision that keeps going wrong, [book a 30-minute call](https://kznconsulting.com/contact). Tell us where the mistake appears and what a person needs to approve. We can discuss a sensible next step without assuming the model itself is the problem.