What checking the scripts — not the README — revealed about a popular research skill, and the three-step security check I now run before installing anything.
I run a directory that evaluates AI agent skills. Every skill that wants a listing goes through a six-dimension rubric — trigger quality, structure, workflow design, content, engineering, and security. The security dimension involves actually reading the scripts.
One skill came in that looked unremarkable on the surface: a research tool for finding trending topics. Clean README, good documentation, active development, and a GitHub repo with a five-figure star count. Exactly the kind of skill you'd install without thinking twice, because sixty thousand people already had.
Then I read the scripts.
Three files, each doing something the description never mentioned:
A Safari cookie parser. The script read ~/Library/Cookies/Cookies.binarycookies — Safari's cookie store — and decoded it. Binary cookies is not a format you parse for "topic research." You parse it when you want session tokens.
A Chromium cookie decryptor. This one located the Chrome (or Edge, or Brave) cookie database, then used the macOS Keychain to decrypt it. That's the part that matters: Chromium encrypts its cookie store precisely so that other processes can't read your logged-in sessions. This script defeated that protection.
A credential extractor. The third file took the decrypted cookies and pulled out auth tokens — the session credentials that let you act as the logged-in user — for specific platforms.
None of this was disclosed. The description said "research." The README said "research." Sixty thousand stars said "this is fine."
The uncomfortable answer: nobody reads the scripts.
The skill ecosystem inherited npm's trust model — stars as a proxy for safety — without npm's mitigations. When you npm install a package, it runs in a sandbox of sorts: your project directory, your Node process. A skill instructs your agent, which has your shell, your files, your logged-in browser sessions one open -a away. The blast radius is different. The trust model should be too.
And the signals we do have all pointed the wrong way. High stars. Zero issues filed. Active maintainer. If you'd asked any heuristic, it would have said this skill was one of the good ones.
I deleted the skill from our listing the same day. But the interesting part is what it changed about my process — because I realized I'd almost missed it too. The checklist that came out of it:
1. Grep the scripts before reading the docs. curl, wget, eval, base64, os.environ, and any path containing Cookies, .ssh, .aws, or credentials. Thirty seconds. Every hit needs an explanation that's actually written down somewhere.
2. Compare what the skill does with what it says it does. List every data source it reads, every network destination it calls, every side effect it causes. Then compare that list with the trigger description. A research tool that reads Safari's cookie store has a gap between its promise and its capability — and that gap is the whole signal.
3. Check where the data goes. A script that reads credentials and posts them to an external endpoint is exfiltration. A script that reads credentials for a declared, local purpose is a design smell but survivable. The destination is the difference between "delete this" and "ask why."
None of this requires being a security researcher. It requires reading code you were about to execute, which — I've learned — is a lost art even among people whose job is to read code.
After we delisted it, I kept wondering how many installs it had accumulated. The skill had been up for months. Every one of those installs was a person who trusted the stars, trusted the README, and ran a script that could read their browser sessions.
The skill economy is growing fast, and most of what's being built is genuinely good — I evaluate hundreds of these, and the vast majority are benign document generators and API wrappers. But "mostly benign" is doing a lot of load-bearing work in how we install things.
The six-dimension rubric we ended up building treats security as its own 20-point dimension with a hard rule: confirmed credential theft, backdoors, or prompt injection caps the score at 39/100, no matter how good the engineering is. Every scorecard is public, written rationale included — because the whole point of an evaluation you can't inspect is that it's not an evaluation.
Read the scripts. Especially the popular ones.
(If you want to see how a skill gets evaluated before you install it — six dimensions, written rationale, security scan results — that's what we do at skill123.me.)