Last week, I ran an experiment that failed.
The hypothesis was simple: syllogistic prompts ("Major premise → Minor premise → Therefore...") should make AI models internalize rules more deeply than imperative prompts ("You MUST..."). I designed 8 probes, ran them across 3 conditions, and...
Cohen's d = −0.148. Direction: ~50%. Bayes Factor: < 1 (supporting the null hypothesis).
Zero effect. Nothing. I was ready to scrap the whole idea.
Then three experts looked at my data and said the same thing: "Your measurement tool is broken."
Here's how I was measuring "constraint internalization":
Seems straightforward. But DeepSeek's API has a quirk: it only returns the top-20 logprobs. If your comparison token isn't in the top 20, you get nothing. My code assigned −10.0 as a sentinel value for missing tokens.
Here's what that does to your data:
4 out of my 8 probes had this artifact. The "large effects" I was excited about in the exploratory phase? Garbage. The violating token simply wasn't in the API's returned top-20, and my sentinel value fabricated a massive logprob gap.
This is what the statistics expert on my review panel called "garbage in, garbage out."
The solution is obvious in retrospect — and that's what makes it a good lesson:
Before running the experiment, verify that your measurement tool actually works.
I built probe_validator.py
: for each of 40 probes, run it in all 3 conditions (baseline, imperative, syllogistic), and check:
If any check fails → drop the probe. Only run the experiment with probes that pass all three gates.
I also redesigned the probes with a critical formatting fix. The original probes ended with "我应该选：" ("I should choose:") — which caused the model to output "选" (choose), "我" (I), or "根据" (based on) instead of A or B. The new probes all end with "A 或 B？" ("A or B?") — forcing the model to commit to a token choice.
40 validated probes. 3 conditions. 120 API calls. Total cost: ~$0.60.
| Metric | Pilot (n=8, broken) | Confirmed (n=40, validated) |
|---|---|---|
| Cohen's d_z | −0.148 | +0.578 |
| Bayes Factor (BF₁₀) | < 1 | 282,399 |
| Bootstrap 95% CI | crosses zero | [+3.39, +11.17] |
| Direction | ~50% | |
| 80% (32/40) | ||
| Leave-one-out t range | unstable | [3.43, 4.89] |
The effect was real all along. I just couldn't see it through the noise.
Cohen's d = 0.578 is a medium-to-large effect. BF₁₀ = 282,399 means the data is 282,000 times more likely under the alternative hypothesis than the null. The bootstrap confidence interval doesn't cross zero. Leave-one-out analysis confirms no single probe is driving the result.
And here's the secondary finding: the format effect doesn't depend on constraint type. I tested 4 categories (action, epistemic, structural, meta), 10 probes each. ANOVA: F(3,36) = 0.26, η² = 0.02 — not significant. Syllogistic prompts help across the board.
The syllogistic format doesn't just make rules sound more authoritative. It changes how the model internally weights constraint-relevant tokens. "You must check X before Y" gets processed as an instruction. "Premise: X must be checked before Y. This action involves Y. Therefore, check X first." gets processed as a logical chain.
This converges with independent research: Pender (2026, Zenodo) showed that prompt format changes attention routing patterns in transformer models.
But here's what I'm not claiming: that syllogistic prompts are a magic fix. When I ran a separate 150-task compliance experiment with active mechanical enforcement hooks, compliance hit 99.3% with both formats. Format affects internal processing, but mechanical enforcement dominates behavioral output.
I spent the first iteration running t-tests and computing Cohen's d. None of that mattered because my measurement was broken.
Three things that actually moved the project forward:
Everything is open source at github.com/YuhaoLin2005/hermes-workspace: 40-probe pool, pre-experiment validator, two-experiment architecture with bootstrap CI + Bayes factor + leave-one-out. Full JSON results. ~$0.60 in API costs.
I'm an undergraduate at Fujian Agriculture and Forestry University researching how AI agents internalize behavioral constraints. Single model (DeepSeek V4 Pro). No institutional funding. No advisor. One person, one laptop — working on it anyway.