Every time I wanted to add an MCP server to Claude Desktop or Cursor, I ended up
with a dozen browser tabs open, trying to answer one question the README never
answers: is this actually safe to give tool-call access to my machine?
MCP (Model Context Protocol) directories exist, but most of them just mirror
GitHub — stars, a README excerpt, a copy-paste config. None of them score
risk. So I built RepoAI, and I want to share the exact methodology, because I think it's more interesting than the product pitch.
Stars measure popularity, not safety. A server with 5,000 stars can still:
None of that shows up in a star count.
computeSecurityScore()
runs 15 checks against each server, sums signed point
values, and clamps the result to 0–100. Two categories:
Repo health (max +101):
| Check | Points | Why |
|---|---|---|
| Official maintainer | +18 / 0 | Vendor-published > random fork |
| Editorially verified | +5 / 0 | A human actually reviewed it |
| Actively maintained | +12 / +6 / 0 | Last commit ≤90d / ≤365d / older |
| Established contributor base | +10 / +5 / 0 | Bus factor: ≥5 / ≥2 / 1 |
| Community adoption | +10 / +5 / 0 | Stars: ≥1000 / ≥100 / fewer |
| Has license | +8 / 0 | Unlicensed = unclear usage terms |
| Reasonable dependency footprint | +5 / +2 / 0 | Smaller supply-chain surface |
| Has homepage/docs | +3 / 0 | |
| Substantial README | +5 / 0 | ≥500 chars |
| Archived | ||
| 0 / −40 | ||
| No future fixes. Ever. |
MCP-specific:
| Check | Points | Why |
|---|---|---|
| Read-only mode available | +15 / 0 | A safer way to run it should exist |
| Supports OAuth | +10 / 0 | Stronger than static tokens |
| Dangerous tools present | 0 / −10 / −25 | Share of tools flagged execute/delete/write |
| Dangerous tools, no safe mode | 0 / −20 | |
| The double penalty — risky and no escape hatch | ||
| Weak or missing auth | 0 / −15 | Anyone with the config file can use it |
Scores land in three bands: 80–100 safe
, 60–79 medium
, 0–59 high
risk.
My first instinct was to let an LLM write the verdict and set the score.
That's a mistake — an AI can be convinced by good marketing copy in a README
just as easily as a human can. So the pipeline splits it:
For a server like qdrant/mcp-server-qdrant
, the pipeline:
medium
risk, mostly held back by no OAuth support and a moderate dependency footprint)213 servers, 36 categories, all re-scraped and re-scored on a rolling basis
so a score reflects last week's commit activity, not last year's.
It's open at repoai.io — MCP is the first layer, Claude Skills and the rest of the AI-tool ecosystem are next, same trust model.
Curious what other people would want a security scorer to check for —
what's bitten you when installing an MCP server?