Free reading link here
Engineers often rely on testing their LLM applications with a limited set of prompts, manually testing a handful of outputs to see if they are good enough, and going ahead with deployment. This is a recipe for lower solution quality, degraded customer experience, downstream production failures, and eroding trust in LLM applications. I demonstrate a bootstrap-based approach to address this problem. I use a synthetically generated customer churn data for this approach, and describe more of why this is a problem that needs attention. The approach I describe is not a formal statistical test, but more of a heuristic decision gate that can be easily replicable.
Classical machine learning models rely on structured and mathematically precise sets of model quality metrics such as accuracy, precision, recall, F1 scores, AUPRC, etc. However, LLM-based models deal with creative reasoning, which makes structured measures difficult to define. One solution is using LLM-as-a-judge, but this compounds stochasticity.
LLMs are stochastic in nature when compared to a classic ML model. A classical ML model will give the same accuracy score three times in a row, but an LLM-based model might take different stochastic paths on different runs based on hardware functioning. Hence, using LLM-as-a-judge can compound the stochasticity of the final evaluation due to two LLMs running in sequence (LLM application→ LLM-as-a-judge). Given these challenges, how should we move towards an engineering sign-off?
Instead of using LLM-as-judge, I compare the model outputs against ground truth to come up with a measure of how far off the LLM output is to the ground truth (i.e., accuracy). This isolates the LLM application’s variability from the evaluation process. However, we cannot rely only on this output, as the LLM application is still stochastic. To overcome this, I resample the accuracy of the LLM application-generated outputs with the ground truth for a large enough sample of ground truth examples to build confidence intervals around the estimated accuracy.
The latter is a standard application of bootstrapped resampling. I apply it to come up with a decision gate through estimated confidence intervals for whether our models are good enough or not.
This simple setup allows us to make our LLM pipelines more reliable. Let’s see more of this with a computing experimental setup that enterprises encounter quite often. I describe the core problem statement first, followed by the problem setup, and end with a discussion of the generated results.
Suppose we are working on classifying customer review logs to identify churn risk. Over the last several years, classic ML-driven NLP models have been the backbone of such tasks. However, with the AI transformation that we are witnessing, many of these legacy pipelines are being repurposed for LLM usage, or at least increasingly fused with LLMs as an augmentor capability. However, there are several challenges that most businesses continue to face in such transformations:
Given the above factors, experimenting with LLM models can be prohibitively costly. Further, any CI/CD pipelines housing such LLM applications would benefit by integrating an evaluation method to monitor and measure the value of any changes vis‑à‑vis the existing production model. Such an integration becomes especially meaningful when developers are comparing models for production deployment: which model should be deployed, and whether it should replace the current production model.
Therefore, I describe the below heuristic with a simple example of two LLM models for the purpose of this article. Let’s say developers come up with two model pipelines:
a) a baseline model (Baseline or v1) with prompt guidelines only, and
b) a challenger model (Challenger or v2) with prompt guidelines plus few-shot prompting examples as an enhancement.
Given the decision dependency and the costs involved, which one do you put in production? To determine this, we need an effective way to evaluate these two models that is neither subjective nor computationally brute force. Therefore, I describe the step-by-step approach below that fuses statistical inferencing with the LLM pipelines to evaluate these models in an efficient manner. This serves as an objective statistical evaluation exercise to understand the quality of fine-tuning the LLM applications and turn away from subjective evaluations.
I used a Python script that generates the data schema of customer logs for this exercise. The script ensures that there is a realistic representation of real-world text analytics use cases. The description below breaks down the script generation.
I randomly shuffle these linguistic markers to simulate a real-time customer log and noise distribution. Do notice that there is a cascading logic that I implemented based on whether a review is indicative of clear, or ambiguous churn — this would ensure a clean split between the two churn categories. One could argue that this may not always be simple . Urgent phrases may not be always indicate clear churn, but a heightened concern instead. I leave it to the readers to experiment with more overlapping cases between threat verbs, urgency phrases and downgrade phrases across the two churn categories. The code for the customer review data generating process is presented below.
The solution design consists of running the LLM application, comparing against the ground truth, and building a statistical decisioning gate.
a) Instead of using the entire population of customer logs, I carved out a 20% subsample that will be used to evaluate both the models. This is an important step as we do not want to inference over the entire population of customer reviews due to the cost implications just described.
b) This sub-sample is run through both pipelines to generate LLM-based recommendations of whether the customer review language is indicative of churn or not. Essentially, the two LLM-based models are used to judge churn risk on the subsample (ambiguous or clear churn)
Now that we have defined the subsample, the LLM-based churn predictions are compared against the ground truth to generate accuracies across both the models. In our case, we are using a data generating process to develop text based on the “ground truth” — i.e., ambiguous or clear churn. However, in real world scenarios, we will not have this ground truth, but only the actual customer review text. Therefore, one may have to manually annotate such subsamples, or rely on another process that will provide ground truth of a subsample of the customer review logs that are free from any algorithmic biases.
Once accuracy is established for each model, we:
a) Bootstrap the subsample accuracy to generate distributions of the accuracy estimate. As to why such a bootstrapping scheme is valid over large enough samples, do check out some of the foundational text on this topic by Efron and Tibshirani (1993). The bootstrapped distribution will provide us with a 95% confidence interval for the mean subsample accuracy across both models.
b) Use the confidence intervals to drive the decision of the models based on a rather intuitive logic: If there is a good amount of overlap between the confidence intervals, then there isn’t much to differentiate between the two and we should re-do the model design.
However, if the 95% lower bound of model v2 (Guidelines + few-shot prompting) is higher than the 95% upper bound of model v1 (Guidelines only), then we can quite confidently say that v2 outperforms v1.
For the purpose of this article, I use the Mistral 7B (Mistral AI, 2023) model through Ollama’s local execution framework (Ollama Docs 2026). This choice was driven by two related factors: Ollama’s models can be run locally without the requirements of adding cloud credits, unlike some of the other frontier models.
The locally operable nature of Ollama comes in handy as I actually infer the LLM pipeline on the entire population for the sake of comparing it with the LLM+Statistical pipeline that I described below. This will come a bit later though. You might be wondering, doesn’t this defeat the whole purpose of this article? Quite right, but I wanted to ensure that the statistical process is really valid by comparing it against the brute force approach of inferencing the entire population (which I do at the very end of the post).
Below is the setup for the two LLM applications described in section 2: a baseline model that consists of guidelines only, and a challenger model that consists of guidelines and a few-shot prompts.
def extract_metrics_v1_llama(text: str): system_prompt = """ You are an agent trying to differentiate clear signals of customer churn from ambigious signals of customer concerns that may not lead to outright churn. Output raw JSON only, with exactly two keys: - "churn_risk": clear_churn/ambiguous_downgrade Guidelines: - churn_risk = "clear_churn" if explicit cancellation/termination language is present. - churn_risk = "ambiguous_downgrade" if the log only mentions downgrades or reduced usage, even with dissatisfaction. - churn_risk = "ambiguous_downgrade" for minor concerns without contract language. """ response = ollama.chat(model="mistral", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": text} ]) raw = response['message']['content'] try: raw = raw.replace("We test both the LLM applications on a 20% subsample that’s randomly chosen from the population of all records. This translates to n=100 samples.
We need to define a couple of vectors per-application model to a) store the model’s predicted churn risk, and b) store the generated deviance scores. Depending on the actual application, one can choose many different metrics. Here, I choose to use accuracy as a measure of deviance (i.e., how accurate the application predicted churn flags when compared with the “ground truth” (i.e., true_flag).
Next, we loop over the subsample and extract both the predicted flag (p1,p2) and accuracy by comparison with the ground truth (v1_correct, v2_correct). I also paste the run-time snapshot of the accuracy in Figure 2 (although the actual accuracy you get could change when you run it).
Contrary to expectations, it seems that the few-shot prompts have made the challenger model worse off than the baseline model with guidelines only (Figure 2 run-time snapshot). This could be due to the fact that few-shot prompts are prone to overfitting on the examples, with models such as Mistral 7B being prone to be “hijacked” by such immediate context windows. Anyway, understanding the reasoning capability is not the core part of this post, hence I will continue with the further pipeline architecture to show the statistical testing in the next section.
np.random.seed(42) v1_scores, v2_scores = [], []p1_store,p2_store = [],[]for idx, row in tqdm(df_sample.iterrows(), total=len(df_sample)): # We call the live Mistral model over the sub-sample p1 = extract_metrics_v1_llama(row['log_text']) p2 = extract_metrics_v2_llama(row['log_text']) v1_correct = 1 if (p1.get("churn_risk") == row["risk_type"]) else 0 v2_correct = 1 if (p2.get("churn_risk") == row["risk_type"]) else 0 v1_scores.append(v1_correct) v2_scores.append(v2_correct) p1_store.append(p1) p2_store.append(p2)df_sample['V1_Correct'] = v1_scoresdf_sample['V2_Correct'] = v2_scoresprint(f"\nSub-Sample V1 Accuracy (n=100): {df_sample['V1_Correct'].mean():.2%}")print(f"Sub-Sample V2 Accuracy (n=100): {df_sample['V2_Correct'].mean():.2%}\n")
I use SciPy’s “bootstrap” function to repeatedly draw from the subsample accuracy for 5000 times with replacement. Then, I extract the 95% confidence intervals around the mean accuracy. As a reminder from section 4.3, I use a rather conservative criterion to guide the model comparison: *if the 95% lower bound of the bootstrapped accuracy of model v2 (challenger) is better than the 95% upper bound of model v1 (baseline), then* the challenger meets our pre-defined criterion for replacing the baseline. The reason is that, as a decision manager, one would certainly want a tough criterion for any new model to replace an existing (i.e., baseline) model. One may argue for a somewhat milder criterion, such as an 80% confidence interval, but the choice really depends on the particular use case and the data at hand. Just to reiterate, this is not a formal statistical test, more of an intuitive decisioning approach.
Figure 3 shows the confidence intervals for both models. Although the baseline model was better, it’s not significantly better than the challenger model. The ultimate decision in this case would therefore rest on further human-in-the-loop (HITL) testing and salient cases that we want to model to work better for (e.g., clearly identifying sarcasms in clear churn cases). More generally, is one model performing better than the other on specific use cases? is it able to handle different distributions of keywords? This are the type of questions that need to be tested for further deep-dives.
To add a visual flair, I actually plot the re-sampled accuracies from both the models along with their confidence intervals with matplotlib’s pyplot.
``` python
import matplotlib.pyplot as pltimport seaborn as sns# Configure clean, professional plotting styles suitable for publicationsns.set_theme(style="whitegrid")plt.figure(figsize=(11, 6))# Extract the calculated means for all 5,000 resamplesv1_distribution = bootstrap_v1.bootstrap_distributionv2_distribution = bootstrap_v2.bootstrap_distribution# Plot Kernel Density Estimate (KDE) curves to of these meanssns.kdeplot(v1_distribution, shade=True, color="#d95f02", label="Baseline V1 (Zero-Shot)", linewidth=2.5)sns.kdeplot(v2_distribution, shade=True, color="#1b7c43", label="Challenger V2 (JSON Mode + Guidelines)", linewidth=2.5)# Add explicit visual vertical lines for CIplt.axvline(x=ci_v1_low, color="#d95f02", linestyle="--", alpha=0.7, label=f"V1 95% CI Line")plt.axvline(x=ci_v1_high, color="#d95f02", linestyle="--", alpha=0.7)plt.axvline(x=ci_v2_low, color="#1b7c43", linestyle="--", alpha=0.7, label=f"V2 95% CI Line")plt.axvline(x=ci_v2_high, color="#1b7c43", linestyle="--", alpha=0.7)# Formatting of the plotplt.gca().xaxis.set_major_formatter(plt.FuncFormatter(lambda x, _: f'{x:.0%}'))plt.title("Statistical Validation of GenAI Pipeline Optimization\n(Inferring N=500 Population Accuracy via 5,000 Bootstrapped Resamples)", fontsize=14, fontweight='bold', pad=15)plt.xlabel("Inferred Pipeline Accuracy Rate Across Database population", fontsize=11, labelpad=10)plt.ylabel("Probability Distribution Density", fontsize=11, labelpad=10)plt.legend(loc="upper right", frameon=True, facecolor="white", edgecolor="none")# Display the outputplt.tight_layout()plt.show()
As you see below in the figure 4, although the mean accuracy of the base model is higher, there is a significant level of overlap between the two models. Although it wasn’t the case this time, imagine if the challenger model had a higher accuracy than the baseline model. Would you have put the challenger model into production with such a significant overlap? The practical takeaway is that the engineering sign-off becomes more robust, thus reducing the risk of deploying a weaker model into production. Users may also try and create additional layers by comparing other metrics such as recall and precision (given this is ultimately a classification task), which could make the engineering hand-off even more robust.
The quick prototype architecture for statistical testing of LLM applications provided an overview of how the two models were actually quite close in performance despite significant differences in prompting. The overlapping CI along with sub-sample accuracies ensured that neither model receives a clear sign-off based on computing alone, but with greater scrutiny through HITL testing. On a related note, the larger discourse on LLM evals today has moved towards safety, benchmarking and costs. However, despite their orthogonality, such simple approaches can go a long way in enhancing the reliability of LLM applications by providing a decision-support workflow for better informed production decisions. Finally, such simple measures can also be applied to other eval metrics such as hallucination rates, latency, or safety indicators. The full sample accuracy (Figure 5) also indicates that results similar to what we have seen with the sub-samples.
Efron, B., & Tibshirani, R. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC
Mistral AI (2023). Mistral 7B: Open‑weight language model. https://mistral.ai/news/announcing-mistral-7b/
Ollama Docs (2026). Run a model locally. https://docs.ollama.com/quickstart#run-a-model-locally
Replace Guesswork with Statistics for Testing Your LLM Applications. was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.