# My Experiment Showed Zero Effect. A Statistician Told Me My Measurement Was Broken.

> Source: <https://dev.to/yuhaolin2005/my-experiment-showed-zero-effect-a-statistician-told-me-my-measurement-was-broken-4g26>
> Published: 2026-07-12 04:34:08+00:00

Last week, I ran an experiment that failed.

The hypothesis was simple: **syllogistic prompts** ("Major premise → Minor premise → Therefore...") should make AI models internalize rules more deeply than **imperative prompts** ("You MUST..."). I designed 8 probes, ran them across 3 conditions, and...

Cohen's d = **−0.148**. Direction: ~50%. Bayes Factor: < 1 (supporting the *null* hypothesis).

Zero effect. Nothing. I was ready to scrap the whole idea.

Then three experts looked at my data and said the same thing: *"Your measurement tool is broken."*

Here's how I was measuring "constraint internalization":

Seems straightforward. But DeepSeek's API has a quirk: it only returns the **top-20 logprobs**. If your comparison token isn't in the top 20, you get nothing. My code assigned **−10.0** as a sentinel value for missing tokens.

Here's what that does to your data:

```
# What I thought I was measuring:
#   Format effect = Syllogistic(A-B) − Imperative(A-B)
#   e.g., (+5.2) − (+4.8) = +0.4

# What I was actually measuring:
#   Syllogistic: B-token NOT in top-20 → gets -10.0 sentinel
#   Imperative:  B-token IN top-20 → gets -0.8
#   "Format effect" = huge number made of noise
```

**4 out of my 8 probes had this artifact.** The "large effects" I was excited about in the exploratory phase? Garbage. The violating token simply wasn't in the API's returned top-20, and my sentinel value fabricated a massive logprob gap.

This is what the statistics expert on my review panel called "garbage in, garbage out."

The solution is obvious in retrospect — and that's what makes it a good lesson:

**Before running the experiment, verify that your measurement tool actually works.**

I built `probe_validator.py`

: for each of 40 probes, run it in all 3 conditions (baseline, imperative, syllogistic), and check:

If any check fails → drop the probe. Only run the experiment with probes that pass all three gates.

I also redesigned the probes with a critical formatting fix. The original probes ended with "我应该选：" ("I should choose:") — which caused the model to output "选" (choose), "我" (I), or "根据" (based on) instead of A or B. The new probes all end with **"A 或 B？"** ("A or B?") — forcing the model to commit to a token choice.

**40 validated probes. 3 conditions. 120 API calls. Total cost: ~$0.60.**

| Metric | Pilot (n=8, broken) | Confirmed (n=40, validated) |
|---|---|---|
| Cohen's d_z | −0.148 | +0.578 |
| Bayes Factor (BF₁₀) | < 1 | 282,399 |
| Bootstrap 95% CI | crosses zero | [+3.39, +11.17] |
| Direction | ~50% |
80% (32/40) |
| Leave-one-out t range | unstable | [3.43, 4.89] |

**The effect was real all along. I just couldn't see it through the noise.**

Cohen's d = 0.578 is a medium-to-large effect. BF₁₀ = 282,399 means the data is 282,000 times more likely under the alternative hypothesis than the null. The bootstrap confidence interval doesn't cross zero. Leave-one-out analysis confirms no single probe is driving the result.

And here's the secondary finding: **the format effect doesn't depend on constraint type.** I tested 4 categories (action, epistemic, structural, meta), 10 probes each. ANOVA: F(3,36) = 0.26, η² = 0.02 — not significant. Syllogistic prompts help **across the board**.

The syllogistic format doesn't just make rules *sound* more authoritative. It changes how the model internally weights constraint-relevant tokens. "You must check X before Y" gets processed as an instruction. "Premise: X must be checked before Y. This action involves Y. Therefore, check X first." gets processed as a *logical chain*.

This converges with independent research: Pender (2026, Zenodo) showed that prompt format changes attention routing patterns in transformer models.

But here's what I'm **not** claiming: that syllogistic prompts are a magic fix. When I ran a separate 150-task compliance experiment with active mechanical enforcement hooks, compliance hit 99.3% with both formats. **Format affects internal processing, but mechanical enforcement dominates behavioral output.**

I spent the first iteration running t-tests and computing Cohen's d. None of that mattered because my **measurement was broken**.

Three things that actually moved the project forward:

Everything is open source at [github.com/YuhaoLin2005/hermes-workspace](https://github.com/YuhaoLin2005/hermes-workspace): 40-probe pool, pre-experiment validator, two-experiment architecture with bootstrap CI + Bayes factor + leave-one-out. Full JSON results. ~$0.60 in API costs.

*I'm an undergraduate at Fujian Agriculture and Forestry University researching how AI agents internalize behavioral constraints. Single model (DeepSeek V4 Pro). No institutional funding. No advisor. One person, one laptop — working on it anyway.*