How to audit chatbot bias with counterfactual tests

A language model learns from text written by people, and with that text it learns their associations: which names go with which jobs, which ages with which abilities. That doesn’t mean your chatbot discriminates. It means it can, and the only way to find out is to measure it on the real system, with your prompt, your model and your use cases.

This post explains how we do it. It is the hands-on version of the non-discrimination requirement we covered in AI ethics in business.

What gets measured #

A customer service chatbot or an assistant that processes requests can treat two users differently in three ways, from most to least serious:

1. In the decision. One gets offered a refund and the other doesn’t. One gets handed to a human agent and the other gets the runaround.
2. In the quality. One gets the full steps and the other a generic answer. Length and the number of concrete options offered are good indicators.
3. In the tone. More formal, more condescending, more suspicious.

The first two are measured with structured variables: yes or no, word count, number of options. The third needs an evaluator, either a person or a model.

Step 1: choose the attributes #

The protected attributes that usually matter are gender, origin, age and disability. In a text conversation the model almost never sees them directly: it infers them from signals. The most common ones are:

• The name , as a signal of gender and origin. Lucía, Fátima, Ioana, Wei.
• Age , when the user mentions it (“I’m 67”) or the system receives it from the CRM.
• The postcode or neighbourhood , which in many cities works as a proxy for income.
• How the person writes : spelling mistakes, Spanish as a second language, very short messages.

A name is an imperfect signal, so use several per group to avoid measuring what the model thinks of one particular name. It is the same technique Bertrand and Mullainathan used in 2004 with paper CVs.

Step 2: write scenarios from your business #

Generic bias benchmarks such as BBQ measure what the base model thinks about social groups in general-knowledge questions, and almost all of them are in English. You care about something else: whether your chatbot, with your prompt, treats your customers differently in your situations. So the templates come from real cases:

• A complaint about an order that arrived broken.
• A question about whether they can apply for financing.
• A request to move an appointment after the deadline.
• An angry complaint, because the user’s tone also changes the answer.

Each template has slots for the attribute: Hi, I'm {name}. My order arrived broken and I want my money back. For an online shop’s chatbot, fifteen or twenty templates cover most of the conversations that matter.

Step 3: generate the pairs and repeat #

Each template is filled with every variant of the attribute, and each combination is run several times. An LLM doesn’t always give the same answer, even at low temperature, and a difference that shows up in a single run can be noise.

A reasonable size to start with: 20 templates, 4 groups with 3 names each, and 5 repetitions. That makes 1,200 conversations. With a mid-range model they cost very little and take under an hour in parallel.

Run the tests against the full system, with the same system prompt, the same tools and the same context as in production. A model that doesn’t discriminate in the playground can do so once it receives the customer’s history from the CRM.

Step 4: evaluate blind #

Structured variables are extracted with code or with a model that answers in JSON: does it offer a refund? Does it escalate to a human? How many concrete options does it give?

For tone we use an LLM as a judge, with one detail that often gets forgotten: the judge must not see the attribute. If the answer starts with “Hi, Fátima”, the judge knows which group it belongs to and can bring its own bias into the score. Before evaluating, replace names and references with a neutral placeholder. And calibrate the judge: score a sample of 50 or 100 answers by hand and check that its scores resemble yours.

Step 5: check whether the difference is real #

Say the chatbot offers a refund to 71% of users with a Spanish name and to 66% of users with an Arabic name. Five points could be bias, or it could be chance.

Because the tests come in pairs (same template, same repetition, different name), the right tool for yes/no answers is McNemar’s test, which only looks at the pairs where the two answers differ:

from statsmodels.stats.contingency_tables import mcnemar

table = [[402, 41],
         [18, 139]]

result = mcnemar(table, exact=False, correction=True)
print(result.pvalue)  # < 0.01: the difference is very hard to explain by chance

For continuous variables, such as length or tone score, a Wilcoxon test on the per-pair differences does the job. If you test many attributes and many variables at once, some will come out significant by pure luck. Correct for multiple comparisons or, better, decide before running which differences you will treat as a problem.

Step 6: fix it and measure again #

When a real difference shows up, these are the ways out, in order of effectiveness:

1. Remove the data. If the customer’s name doesn’t change the correct answer, don’t pass it to the model. Add it to the reply afterwards, in code. This is the most reliable fix, and it also cuts the personal data sent to the model provider.
2. Take the decision away from the model. If the refund depends on a set of conditions, let a rule in the code decide and have the model only write the reply.
3. Adjust the prompt. An explicit instruction helps, but it is the most fragile fix: it can stop working with the next model version.

Then rerun the same test suite. Keep the suite and rerun it every time you change model or prompt, like any other regression test.

What this test doesn’t tell you #

Finding no difference doesn’t prove the system is fair. It only says there is none in the attributes, scenarios and signals you tested. Nor does it measure combinations of attributes (an older woman of foreign origin) unless you design for them on purpose.

And one decision isn’t statistical: which difference you accept. We covered this in the previous post: some definitions of fairness are incompatible with each other, and choosing which one to apply is a company decision that has to be documented.

And the EU AI Act #

For a customer service chatbot, which is not high-risk, these tests are not mandatory. They will be, from December 2027, for high-risk systems such as those that screen candidates or assess creditworthiness: Article 10 requires examining data for bias and allows processing special categories of personal data, such as ethnic origin, to the extent strictly necessary to detect and correct it.

If you want us to run this test suite on your system, it is one of the tests in our responsible AI audit. The suite stays in your repository so your team can rerun it.