{"slug": "i-built-an-aws-agent-that-scores-any-github-repo-before-you-clone-it", "title": "I built an AWS agent that scores any GitHub repo before you clone it", "summary": "A developer built RepoPilot, an AWS-based agent that scores any public GitHub repository from 0 to 100 across safety, trust, maintenance, maturity and documentation before a user clones it. The tool was motivated by Island research on the FakeGit campaign, which involved roughly 7,600 malicious repositories from about 6,600 fake profiles and more than 14 million downloads, including over 800 posing as AI skills or MCP servers. The developer says the agent also flags malicious install steps and can profile a publisher's account, and notes that pointing it at ollama/ollama yields a D grade.", "body_md": "If you use an AI coding assistant, it has told you to clone a repo or run an install command. You probably ran it. Here is why that habit stopped being safe in 2026.\n\nSecurity researchers at Island found a campaign they call FakeGit: about **7,600 malicious GitHub repositories** from roughly **6,600 fake profiles**, with **more than 14 million downloads** across the campaign's release assets. Over **800 of them posed as AI skills or MCP servers**, the exact pieces we are wiring into our agents right now.\n\nThen comes the part that got my attention. Island calls it AgentBaiting. In their tests, **Claude Code, Gemini, and ChatGPT each surfaced a malicious repo on their own, with no link ever handed to them.** Ask an agent for a \"free Walmart MCP server\" and it can return an attacker's repo as its top pick, reading the malicious README as if it were normal docs. The bait is not just a convincing README anymore. The bait is your own agent, vouching for a repo it never inspected.\n\nSo I built **RepoPilot**: an agent you point at any public GitHub repo. In a minute or two it tells you whether to trust it, scores it from 0 to 100, flags install steps built to steal your credentials, and then hands you one clean path from clone to a working first run.\n\nLive demo, no setup: [https://simplynadaf.github.io/repopilot-your-first-run-agent/](https://simplynadaf.github.io/repopilot-your-first-run-agent/)\n\n**🏗️ Built for the [AWS Builder Center \"Build an Agent\" Weekend Challenge](https://builder.aws.com/content/3KSGqk5omLzBXfpe6VIOritaAQn/score-any-github-repo-0-100-before-you-clone-it-an-aws-ai-agent) · #agents**\n\n⭐ If RepoPilot saved you from a painful first run, give it a star. It helps other builders find it.\n\n[The Problem](https://github.com/simplynadaf/repopilot-your-first-run-agent#-the-problem) • [What It Does](https://github.com/simplynadaf/repopilot-your-first-run-agent#-what-it-does) • [Architecture](https://github.com/simplynadaf/repopilot-your-first-run-agent#-how-it-works) • [Getting Started](https://github.com/simplynadaf/repopilot-your-first-run-agent#-getting-started) • [FAQ](https://github.com/simplynadaf/repopilot-your-first-run-agent#-faq)\n\n**📖 Table of Contents**\n\n**Jump to:** Why this is urgent · What it does · How I built it on AWS · Where it is weak · Proof it works\n\nA year ago, \"check the repo before you clone it\" was advice for the paranoid. In 2026 it is table stakes, because the attack moved to where we stopped looking:\n\nThe through-line: the dangerous part is rarely the code you read. It is the setup step you run on faith. The `curl ... | bash`, the postinstall hook, the \"download this ZIP and run it.\" And now an agent is often the one recommending it.\n\nRepoPilot puts a short check between \"a bot told me to\" and \"I ran it.\"\n\nGive it `owner/repo` and it returns a **0 to 100 score**, like a credit score, across five weighted dimensions. Every point traces back to a signal in the repo's metadata and files, not to a vibe:\n\n| Dimension | Weight | What it reads | \n|---|---|---|\n| **Safety** | 30% | Setup steps for malware patterns: `curl \\ | \n| **Trust** | 25% | Stars-vs-age ratio, fork and contributor signals, the fake-star pattern | \n| **Maintenance** | 20% | Recent commits, open-vs-closed issues, release cadence | \n| **Maturity** | 15% | Age, contributor count, overall footprint | \n| **Documentation** | 10% | README depth, license, examples | \n\nTwo dimensions carry the weight against the 2026 threats:\n\nThen it helps you run the thing: one ordered path from clone to running, the tool versions you need, and when a command fails it turns the error into a plain-language cause plus one next step.\n\nType a **username** instead and it profiles the developer: footprint, main languages, top repos by stars. Those ~6,600 fake profiles are the other half of the attack. Vet the person who published the code, not only the code itself.\n\nPoint RepoPilot at {% raw %}`ollama/ollama`, a tool millions install, and it drops to a **D**. Ollama is a legitimate project, so this is not a malware verdict. The flag is about its official install: `curl -fsSL https://ollama.com/install.sh | sh`, which pipes a remote script straight into your shell. RepoPilot does not cry wolf. It says: read this before you run it. Even trusted repos have setup steps worth a second look, and an honest tool admits that.\n\nThe stack is small on purpose:\n\nThat split is the whole point, and it is a tradeoff I would make again. The 0 to 100 number, the critical flags, the version detection, all of it is deterministic Python I can unit-test with no AWS in the loop. The LLM owns tone, not truth. So I can prove the score is right, and the model cannot talk a repo into looking safe.\n\nTwo things I would tell anyone building a Bedrock agent on Lambda:\n\n`converse` call told me Nova Pro was reachable in about half a second. Far cheaper than debugging an `AccessDenied` after you have built the whole thing. I picked Nova Pro over Claude on Bedrock for one reason: this job is narration, not reasoning, since the Python already decided the score. For narration, Nova Pro is cheaper per token and fast enough, so paying for a frontier model here would buy nothing.`requests` into the Lambda zip and hit a wall: its bundled `urllib3` imported something Python 3.12 had removed. I rewrote the GitHub fetcher on the standard library. The package went from 400KB to 13KB and the error was gone.\nThe danger scanner needed tuning too. My first version flagged Flask as unsafe because its docs mention `127.0.0.1`, so I taught the scanner to ignore localhost and private ranges while still catching public-IP installers. A false positive on a trusted framework costs you more trust than a missed edge case does.\n\nOn cost: because the scoring is plain Python and the narration is one short Nova Pro call, a scan is cheap to run on your own Lambda. There is no per-seat subscription and no third-party API metering every repo you check. You pay AWS for your own usage and nothing else, which is the point. A hosted SaaS checker bills you per seat whether you scan one repo or none.\n\nThe whole thing has **25 tests**, including 200 randomised repos that prove the weighted score never breaks its own math and that a critical flag always caps the grade.\n\nA security tool that only says \"no\" is a tool people route around. So I spent my care on the moment a new contributor usually quits: the first failed command.\n\nWhen your first install breaks, RepoPilot does not throw the stack trace back at you. It says something like:\n\n\"A dependency the code imports has not been installed. Run the install step again in the right environment, and activate your virtualenv first. You are close. This is a common first-run snag, not something you broke.\"\n\nCause, next step, and a little reassurance. That is the gap between a tool that reports problems and one that gets you past them.\n\nThe web UI has two modes so it never dead-ends. **Demo mode** is pre-configured and instant, so anyone can click around with zero setup. **Live mode** calls the deployed agent on AWS, and if a live call stalls it falls back to demo data instead of spinning forever.\n\nAgainst the Flask repo, the deployed Lambda fetched 236 files, scored it **98/100 (grade A)**, with safety, trust, maintenance, and maturity at 100 and documentation at 85, detected the Python stack, and read `python >=3.11` from the actual `pyproject.toml`.\n\n`langchain-ai/langchain` scores a clean `ollama/ollama` drops to a `curl | sh` install.`@torvalds` resolves to Linus Torvalds, about 263,000 stars across his repos.\nEvery number traces back to a tool output, never the model's imagination. These four are the repos I have run end to end against the deployed agent so far: three clean, one honest caution. The point of shipping it is to grow that list with repos I did not pick.\n\nAn honest security tool should tell you its blind spots, so here are RepoPilot's:\n\nRepoPilot is a thirty-second gut check that catches the obvious and the lazy, which covers most of this campaign. For code you are about to run with real access, read it yourself too.\n\nA closed, hosted \"repo checker\" would have been a worse fit for this problem, for four reasons that compound.\n\nYour scans stay private. The repos and usernames you are suspicious about are a signal in themselves, and RepoPilot runs on your own AWS account or fully local via the CLI, so nothing goes to a third party. The cost model follows from that: open weights and your own Lambda mean you can screen every repo your agent suggests without a per-check fee or a meter running in someone else's cloud. When a new attack pattern shows up this month, you add it to the pure-Python rules yourself instead of waiting on a vendor's roadmap while the threat is live. And none of it locks you in, since you can swap Nova Pro for any model the narration layer supports and the scoring engine does not care.\n\nWhen the threat is open-source supply chains, the defense should be open too, so anyone can read exactly how the score is computed and decide whether to trust it.\n\nHere is what I would ask of you: next time your AI assistant tells you to clone or install something, run that `owner/repo` through RepoPilot first. Then drop it in the comments with the score it got. I want to see which \"trusted\" repos fail, and which fakes we catch together.\n\nWhat repo should RepoPilot inspect next?\n\n*Follow me for more on AWS architecture, DevOps, and AI Infrastructure:*\n\n[Portfolio](https://sarvarnadaf.com) | [LinkedIn](https://www.linkedin.com/in/sarvar04/) | [Dev.to](https://dev.to/sarvar_04) | [YouTube](https://www.youtube.com/@sarvar-nadaf) | [Email](mailto:simplynadaf@gmail.com) | [AWS Builder Center](https://builder.aws.com/community/@sarvar) | [X](https://x.com/SarvarN_04)", "url": "https://wpnews.pro/news/i-built-an-aws-agent-that-scores-any-github-repo-before-you-clone-it", "canonical_source": "https://dev.to/aws-builders/i-built-an-aws-agent-that-scores-any-github-repo-before-you-clone-it-4omp", "published_at": "2026-10-19 12:03:27+00:00", "updated_at": "2026-10-19 12:14:43.647778+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-safety", "developer-tools", "ai-infrastructure"], "entities": ["RepoPilot", "AWS", "GitHub", "Island", "Claude Code", "Gemini", "ChatGPT", "Ollama"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-built-an-aws-agent-that-scores-any-github-repo-before-you-clone-it", "markdown": "https://wpnews.pro/news/i-built-an-aws-agent-that-scores-any-github-repo-before-you-clone-it.md", "text": "https://wpnews.pro/news/i-built-an-aws-agent-that-scores-any-github-repo-before-you-clone-it.txt", "jsonld": "https://wpnews.pro/news/i-built-an-aws-agent-that-scores-any-github-repo-before-you-clone-it.jsonld"}}