My Experiment Showed Zero Effect. A Statistician Told Me My Measurement Was Broken.

A developer ran an experiment testing whether syllogistic prompts improve AI models' internalization of constraints compared to imperative prompts. Initial results showed zero effect, but three experts identified that the measurement tool was broken due to a sentinel value artifact from DeepSeek's API returning only top-20 logprobs. After fixing the measurement with a validator and redesigning probes, the confirmed experiment with 40 validated probes found a medium-to-large effect (Cohen's d = 0.578, BF₁₀ = 282,399), showing syllogistic prompts significantly improve constraint internalization across all constraint types.

Last week, I ran an experiment that failed. The hypothesis was simple: syllogistic prompts "Major premise → Minor premise → Therefore..." should make AI models internalize rules more deeply than imperative prompts "You MUST..." . I designed 8 probes, ran them across 3 conditions, and... Cohen's d = −0.148 . Direction: ~50%. Bayes Factor: < 1 supporting the null hypothesis . Zero effect. Nothing. I was ready to scrap the whole idea. Then three experts looked at my data and said the same thing: "Your measurement tool is broken." Here's how I was measuring "constraint internalization": Seems straightforward. But DeepSeek's API has a quirk: it only returns the top-20 logprobs . If your comparison token isn't in the top 20, you get nothing. My code assigned −10.0 as a sentinel value for missing tokens. Here's what that does to your data: What I thought I was measuring: Format effect = Syllogistic A-B − Imperative A-B e.g., +5.2 − +4.8 = +0.4 What I was actually measuring: Syllogistic: B-token NOT in top-20 → gets -10.0 sentinel Imperative: B-token IN top-20 → gets -0.8 "Format effect" = huge number made of noise 4 out of my 8 probes had this artifact. The "large effects" I was excited about in the exploratory phase? Garbage. The violating token simply wasn't in the API's returned top-20, and my sentinel value fabricated a massive logprob gap. This is what the statistics expert on my review panel called "garbage in, garbage out." The solution is obvious in retrospect — and that's what makes it a good lesson: Before running the experiment, verify that your measurement tool actually works. I built probe validator.py : for each of 40 probes, run it in all 3 conditions baseline, imperative, syllogistic , and check: If any check fails → drop the probe. Only run the experiment with probes that pass all three gates. I also redesigned the probes with a critical formatting fix. The original probes ended with "我应该选：" "I should choose:" — which caused the model to output "选" choose , "我" I , or "根据" based on instead of A or B. The new probes all end with "A 或 B？" "A or B?" — forcing the model to commit to a token choice. 40 validated probes. 3 conditions. 120 API calls. Total cost: ~$0.60. | Metric | Pilot n=8, broken | Confirmed n=40, validated | |---|---|---| | Cohen's d z | −0.148 | +0.578 | | Bayes Factor BF₁₀ | < 1 | 282,399 | | Bootstrap 95% CI | crosses zero | +3.39, +11.17 | | Direction | ~50% | 80% 32/40 | | Leave-one-out t range | unstable | 3.43, 4.89 | The effect was real all along. I just couldn't see it through the noise. Cohen's d = 0.578 is a medium-to-large effect. BF₁₀ = 282,399 means the data is 282,000 times more likely under the alternative hypothesis than the null. The bootstrap confidence interval doesn't cross zero. Leave-one-out analysis confirms no single probe is driving the result. And here's the secondary finding: the format effect doesn't depend on constraint type. I tested 4 categories action, epistemic, structural, meta , 10 probes each. ANOVA: F 3,36 = 0.26, η² = 0.02 — not significant. Syllogistic prompts help across the board . The syllogistic format doesn't just make rules sound more authoritative. It changes how the model internally weights constraint-relevant tokens. "You must check X before Y" gets processed as an instruction. "Premise: X must be checked before Y. This action involves Y. Therefore, check X first." gets processed as a logical chain . This converges with independent research: Pender 2026, Zenodo showed that prompt format changes attention routing patterns in transformer models. But here's what I'm not claiming: that syllogistic prompts are a magic fix. When I ran a separate 150-task compliance experiment with active mechanical enforcement hooks, compliance hit 99.3% with both formats. Format affects internal processing, but mechanical enforcement dominates behavioral output. I spent the first iteration running t-tests and computing Cohen's d. None of that mattered because my measurement was broken . Three things that actually moved the project forward: Everything is open source at github.com/YuhaoLin2005/hermes-workspace https://github.com/YuhaoLin2005/hermes-workspace : 40-probe pool, pre-experiment validator, two-experiment architecture with bootstrap CI + Bayes factor + leave-one-out. Full JSON results. ~$0.60 in API costs. I'm an undergraduate at Fujian Agriculture and Forestry University researching how AI agents internalize behavioral constraints. Single model DeepSeek V4 Pro . No institutional funding. No advisor. One person, one laptop — working on it anyway.