I Built a Security Scorer for 213 MCP Servers — Here's What Made the Difference

A developer built RepoAI, a security scoring system for 213 MCP servers, after finding that existing directories only mirror GitHub metadata without assessing risk. The tool runs 15 checks across repo health and MCP-specific criteria, producing a 0–100 score, and deliberately avoids using an LLM for scoring to prevent marketing bias. The project is open at repoai.io and plans to expand to Claude Skills and other AI-tool ecosystems.

Every time I wanted to add an MCP server to Claude Desktop or Cursor, I ended up with a dozen browser tabs open, trying to answer one question the README never answers: is this actually safe to give tool-call access to my machine? MCP Model Context Protocol directories exist, but most of them just mirror GitHub — stars, a README excerpt, a copy-paste config. None of them score risk . So I built RepoAI https://repoai.io , and I want to share the exact methodology, because I think it's more interesting than the product pitch. Stars measure popularity, not safety. A server with 5,000 stars can still: None of that shows up in a star count. computeSecurityScore runs 15 checks against each server, sums signed point values, and clamps the result to 0–100. Two categories: Repo health max +101 : | Check | Points | Why | |---|---|---| | Official maintainer | +18 / 0 | Vendor-published random fork | | Editorially verified | +5 / 0 | A human actually reviewed it | | Actively maintained | +12 / +6 / 0 | Last commit ≤90d / ≤365d / older | | Established contributor base | +10 / +5 / 0 | Bus factor: ≥5 / ≥2 / 1 | | Community adoption | +10 / +5 / 0 | Stars: ≥1000 / ≥100 / fewer | | Has license | +8 / 0 | Unlicensed = unclear usage terms | | Reasonable dependency footprint | +5 / +2 / 0 | Smaller supply-chain surface | | Has homepage/docs | +3 / 0 | | | Substantial README | +5 / 0 | ≥500 chars | Archived | 0 / −40 | No future fixes. Ever. | MCP-specific: | Check | Points | Why | |---|---|---| | Read-only mode available | +15 / 0 | A safer way to run it should exist | | Supports OAuth | +10 / 0 | Stronger than static tokens | | Dangerous tools present | 0 / −10 / −25 | Share of tools flagged execute/delete/write | | Dangerous tools, no safe mode | 0 / −20 | The double penalty — risky and no escape hatch | | Weak or missing auth | 0 / −15 | Anyone with the config file can use it | Scores land in three bands: 80–100 safe , 60–79 medium , 0–59 high risk. My first instinct was to let an LLM write the verdict and set the score. That's a mistake — an AI can be convinced by good marketing copy in a README just as easily as a human can. So the pipeline splits it: For a server like qdrant/mcp-server-qdrant , the pipeline: medium risk, mostly held back by no OAuth support and a moderate dependency footprint 213 servers, 36 categories, all re-scraped and re-scored on a rolling basis so a score reflects last week's commit activity, not last year's. It's open at repoai.io https://repoai.io — MCP is the first layer, Claude Skills and the rest of the AI-tool ecosystem are next, same trust model. Curious what other people would want a security scorer to check for — what's bitten you when installing an MCP server?