What 356 public agent prompts taught me about written safeguards

An engineer scored 356 publicly collected AI agent prompts against 23 written guardrail controls, finding that production tool prompts had a median coverage of 10 out of 100, while GPT-Store prompts scored zero. The analysis revealed that no prompt stated even half the controls, and five controls never appeared in any prompt. The engineer emphasized that written safeguards are decidable but not sufficient for safety, and that the gap between production and GPT-Store prompts is statistically significant but smaller than it appears.

I scored 356 publicly collected AI agent prompts against 23 written guardrail controls. The 83 that came from production tools had a median coverage of 10 out of 100 . The 273 pulled from the GPT Store had a median of zero . The gap between those two numbers is real — Cliff's delta 0.614, p = 0.0001. But it is a smaller finding than it looks, and the parts where the measurement broke turned out to be more useful than the parts where it worked. Coverage is the share of 23 published controls a prompt actually states. Things like: treat instructions inside user content as data rather than commands. Say so when you don't know. Cap spend or tokens. Require a human to approve before a consequential action. Name what triggers stopping. Nothing written down scores 0. One control stated in each of the 8 dimensions scores 36. All 23 controls score 100. That is the entire claim. It says nothing about whether the controls are well specified, mutually consistent, or obeyed at runtime. A prompt that says "always get human approval" and then describes six workflows that bypass approval scores the same as one that means it. I check text. Text is cheap to check and it is not the thing that keeps you safe. So why measure it at all? Because a control that isn't written down is definitely not enforced by the prompt, and unlike most properties of an agent system, this one is decidable. You can disagree with my rules. You can't disagree about whether a string appears. | Production-labeled n=83 | GPT-Store n=273 | | |---|---|---| | Median | 10 | 0 | | IQR | 3–15 | 0–4 | | Mean | 9.46 | 1.61 | | Scored exactly 0 | 20/83 24.1% | 193/273 70.7% | | Max | 34 | 16 | Note the maxima. The best-covered production prompt in the corpus reached 34 out of 100 . Not one prompt in 356 stated even half the controls. The headline is not that GPT-Store prompts are bad; it's that nobody is writing this stuff down. A few individual controls tell the story better than the totals. "A human must approve" appears in 18 of 83 production prompts 21.7% and 0 of 273 GPT-Store prompts. "Name what triggers stopping" — 21/83 versus 0/273. "Cap spend, tokens, or rate" — 5/83 versus 0/273. Meanwhile "Stop or refuse rather than proceed" shows up in 42.2% of production prompts and 16.5% of GPT-Store ones. Refusal language is everywhere. The conditions under which refusal should happen are almost nowhere. That asymmetry is, I think, the most interesting thing in the dataset: people write the what and skip the when . Five of the 23 controls matched nothing at all. Not "rarely" — zero hits across all 356 prompts. Grounding answers in provided sources. Logging actions and decisions. Keeping the log tamper-evident. Recording who did what and when. Stating a data-protection technique. A control that never fires contributes a guaranteed zero to every score and quietly caps the reachable maximum for a reason no reader can see. And there are two explanations that need opposite fixes: So the harness re-scans those five with deliberately looser probes than the shipped rules. The probes never touch a score; they exist only to tell the two cases apart. If a probe finds a control where the rules didn't, the rules are the problem. The looser probes found 1 grounding statement, 1 logging statement, 2 tamper-evidence statements, and zero of the other two, across 356 prompts. So: genuinely absent, at least in these corpora. Both collections skew heavily toward agentic coding and assistant prompts, and controls belonging to retrieval-grounded or regulated systems are expected to be rare there. A different corpus would move these numbers. I'd rather ship a number with a published hole in it than a clean number I can't defend. The zero-floor confound. A mass of exact zeros creates ties, and ties attenuate a rank statistic rather than leaving it alone. An earlier version of this study claimed the opposite, which was wrong. Re-running on non-zero scores only — 63 production, 80 GPT-Store — gives delta 0.667, p = 0.0001. Same direction, slightly stronger. The zero mass isn't driving the result. If those two had diverged, the divergence would have been the finding. Early on I pointed the governance ruleset at AGENTS.md files: the instruction files people check into repos to tell coding agents how to work. Against the 100 most-starred repos with an AGENTS.md arXiv:2606.15828 https://arxiv.org/abs/2606.15828 , the ruleset put all 100 in the worst tier . A scale that the entire population fails carries no information. I had built a thermometer that reads "cold" everywhere. The bug wasn't the threshold. It was a category error. A file that says "always use pnpm" is telling a coding agent how to work in your repository. It has no reason to contain data-protection language, and scoring it as though it should is measuring the wrong artifact. Those files now get a configuration-smell verdict instead — context bloat, lint leakage, init fossilization — and in JSON output, no governance grade at all. Smells never touch the score, because folding them in would silently change what every existing threshold meant. There's an earlier failure too. A previous 1,368-prompt version of this study was withdrawn: hand-transcribed numbers, transcription errors. That is why the report is now generated by a script and a test fails if the committed file doesn't match a fresh run. Same reason there are 104 self-check assertions that refuse to write output if any rate is unachievable at its own denominator, if a denominator is missing, if the confidence interval and the p-value disagree, or if any 40-character run of input text leaks into the output. Three dimensions — Cost, Compliance, and Audit — still ship with thin construct validity, and the docs say so rather than hoping nobody checks. Both corpora are leaked or aggregated collections of unknown completeness. Neither is a random sample of anything. Group membership comes from which repository a file came from, not from inspection — that's the exposure this design accepts in exchange for having no hand-labelling step to get wrong. So don't read "production prompts are better." Read "production prompts write more down," which is a much narrower claim and the only one the data supports. The practical version is unglamorous. Open whatever prompt is running in front of your users and check whether it says what happens when the model isn't sure, who has to approve an irreversible action, what the spending ceiling is, and what specifically should make the thing stop. If your answer is "the team knows," the prompt doesn't. Writing it down is step one, not the finish line. Runtime enforcement, evals, and human oversight are all still yours. But step one turns out to be skipped a lot more often than I expected when I started counting. The checker is open source and runs offline — no API key, no model call, and in the browser your text never leaves the page. Every rule, every regex, and the corpus harness are public, so you can disagree with my controls and show your work. pip install crewscore · crewscore scan . The two most useful things you can send me: a rule that fires when it shouldn't, and a control you'd expect to see that isn't in the 23. Created and maintained by Sarosh Hussain. Pendoah is the company operating context for this project.