# I Built an AI to Judge AI Projects — Here’s What It Looks For > Source: > Published: 2026-10-15 08:58:31+00:00 *There is a surprisingly difficult question hiding behind almost every technical portfolio:* ***how good is this project, actually?*** Not how polished the README looks, how many stars it received, whether the repository name contains *LLM*, *Agent*, or *Computer Vision*, or whether its author wrote *AI Engineer* on LinkedIn. I mean something more specific: **what does the implementation actually demonstrate?** ***Both are legitimately Transformer projects.*** *But they are not remotely the same piece of work.* That problem became one of the most interesting parts of building [AI-Atlas](https://getaiatlas.com/?ref=medium2), the platform I have been developing to turn public GitHub work into an explainable map of demonstrated AI skills. AI-Atlas currently maps projects across **Machine Learning, Computer Vision, Data Science, NLP & LLMs, AI Agents, and MLOps**, but the domain classification itself is only the beginning. Once you decide to evaluate the work itself, you inherit a much harder problem: defining what *good AI work* actually means, distinguishing implementation from imported functionality, separating originality from complexity, and producing a judgment that is specific enough for the developer to inspect and disagree with. That last part became especially important to me. ***If an AI is going to judge someone’s work, “the model said so” is not a good enough explanation.*** A repository’s subject tells us surprisingly little about its quality. Take something as familiar as a **ResNet implementation**. One developer might import a pretrained architecture from **torchvision**, replace its classification head, fine-tune it on a dataset, and produce a perfectly functional project. Another might implement residual blocks independently, reproduce **several ResNet variants**, compare initialization strategies, build a reusable training pipeline, investigate how shortcut connections affect optimization, and document every experiment. Both repositories belong under **Computer Vision.** Their topic is nearly identical. **The evidence of technical ability is not.** That is why AI-Atlas does not evaluate a project with one vague question like *“Is this impressive?”* *Each relevant repository is examined across seven dimensions:* ***Technical Depth, Conceptual Difficulty, Originality/Modification, Completeness, Practical/Research Usefulness, Documentation & Educational Value, and Engineering & Reproducibility.*** Those dimensions intentionally measure different things. A project **can attempt a difficult idea** but remain unfinished. It can be **exceptionally well engineered** without containing an original research contribution. A **from-scratch educational implementation might be less novel than a new architecture**, while still demonstrating considerably more understanding than a thin wrapper around an API. Even a sophisticated experiment becomes harder to trust if there is no reliable way to determine what configuration produced its result. Now imagine another repository with the same title containing a short notebook that instantiates **an existing Vision Transformer implementation and fine-tunes it**. ***Both contain valid AI work,*** *but the first exposes substantially more evidence about the developer's understanding of the architecture, training process, experimental decisions, and engineering choices.* This is the distinction I wanted the evaluator to learn to make. This was probably the most important design **constraint** in the entire system. If the explanation is simply *“a language model analyzed the repository”*, the number is **almost meaningless**. We would have replaced one shallow signal with another, except the new one looks more authoritative. AI-Atlas therefore separates **interpretation** from **scoring**. The evaluator reads the relevant implementation and returns judgments across the rubric together with its reasoning and supporting repository evidence. AI-Atlas then **computes Project Merit through versioned scoring rules** rather than allowing the language model to invent the final number directly. That separation sounds subtle, but it matters. If I later decide that **Technical Depth** should contribute differently to **Project Merit**, that is a scoring-methodology change. If the evaluator **incorrectly interprets** a custom optimizer as ordinary library usage, that is an evidence-evaluation problem. Those are **fundamentally different failure modes**, and keeping them separate makes the system much easier to debug and criticize. The same principle applies to the public methodology. [AI-Atlas](https://getaiatlas.com/?ref=medium2) exposes how repositories are discovered, understood, mapped, and explained, and the live site explicitly describes the central rule as **“implementation, not labels,” “depth over repetition,” and “claims back to files.”** This also means a high-level score is **not intended to be the final destination.** The interesting path is in the opposite direction: from the profile score to the domain, from the domain to the project, from the project to the rubric judgment, and eventually back toward the code that caused the judgment. ***The score becomes useful only when you can trace it back to something real.*** *One of the easiest ways to make an* ***AI portfolio evaluator*** *appear impressive would be to discover AI expertise everywhere. That is also one of the easiest ways to destroy its usefulness.* Real GitHub profiles contain **forks, abandoned experiments, personal websites, coursework, profile READMEs, utilities, duplicate repositories, API wrappers,** and **projects that simply have nothing to do with artificial intelligence.** Treating all of them as positive evidence would make larger and noisier profiles appear stronger almost automatically. *The public site summarizes this as a* ***Discover → Understand → Map → Explain*** *pipeline: first remove obvious noise, then inspect what was actually built, then place genuine evidence into the appropriate domains.* There is also a security reason for keeping this process conservative. Repository contents are treated as **untrusted data**. AI-Atlas reads public code as text, but it does not clone a stranger’s project and start executing installation scripts, Dockerfiles, tests, or arbitrary shell commands. The live trust section explicitly states that code is **read, never run**. That means the evaluator is **working from inspectable evidence** rather than trusting the repository to demonstrate itself through execution. **Popularity is deliberately kept separate too** GitHub stars can indicate adoption, and forks can tell us that a project has attracted real usage, but those numbers answer a different question from technical quality. **A simple developer tool** can accumulate thousands of stars, while **a difficult research reproduction** written by one student can receive five. AI-Atlas therefore keeps the technical rubric based on the project itself, with only a small capped adoption adjustment for projects that cross meaningful usage thresholds. This matters particularly for students and independent developers. **If popularity dominates the score, the evaluator is measuring audience size rather than engineering ability.** ***Technical Depth*** *asks how much meaningful implementation exists beneath the project’s headline.* Using **PyTorch, Hugging Face**, or **scikit-learn** does not make a project shallow; serious engineering almost always depends on libraries. The question is whether the developer **made substantial technical decisions themselves.** A repository containing custom model components, data pipelines, evaluation infrastructure, training logic, checkpointing, and careful experimental code demonstrates **a different level of implementation depth** from a notebook that calls a ready-made pipeline. ***Conceptual Difficulty*** *concerns the problem itself.* Implementing **linear regression, reproducing a Transformer, building a multimodal retrieval system,** and **designing a distributed agent runtime** do not demand the same level of conceptual understanding. At the same time, complexity should not become a reward by itself. A difficult idea implemented incompletely **should not receive full credit** merely because the README contains advanced terminology. ***Originality/Modification*** *asks what the developer actually contributed beyond reproduction.* That can mean **changing an architecture, developing a custom loss, testing an unusual training strategy, combining methods in a non-trivial way**, or **designing experiments around a genuine research question.** Originality does not require producing a publishable paper. A careful reproduction can still be valuable. But *reproducing* and *modifying* demonstrate different abilities, so the evaluator should distinguish them. ***Completeness*** *asks whether the repository forms a coherent project rather than a collection of promising fragments.* A model implementation with no training entry point, no evaluation, and no instructions may demonstrate technical understanding, but it is still **less complete** than a repository where the components form an end-to-end experiment. ***Practical/Research Usefulness*** *asks whether the project accomplishes something meaningful.* Sometimes usefulness means supporting a real application. Sometimes it means testing a research hypothesis or making an important paper easier to understand. I deliberately **do not equate usefulness with commercial value.** ***Documentation & Educational Value*** *concerns whether another technical reader can understand the decisions behind the code.* Why was this architecture chosen? Which assumptions were made? What failed? How are the reported results reproduced? Good documentation **does not substitute for technical depth**, but it makes technical depth inspectable. *Finally,* ***Engineering & Reproducibility*** *asks whether somebody else can reasonably verify or continue the work.* Configuration files, clear dependencies, random seeds, tests, evaluation scripts, dataset instructions, checkpoint conventions, and stored experiment settings all make a repository considerably stronger. Not every project requires every item, but their presence communicates something important: **the developer thought beyond making the code work once on their own machine.** That is ultimately what the evaluator is trying to reconstruct. Not a list of technologies, but the shape of the engineering demonstrated through their use. *Once I had a scoring system, I* ***became increasingly interested*** *in a different question.* Suppose the evaluator decides that a project is technically strong but its **Engineering & Reproducibility** is considerably weaker than the rest. What should the developer actually do with that information? The least useful answer would be *“raise the score.”* That turns the entire system into a game. A much better question is: **what real change to the repository would make the project better?** That is the idea behind the owner-only **Start here** recommendation. It does not simply select the repository with the lowest score. Instead, AI-Atlas looks for the project where **the available rubric headroom represents the most meaningful lever** for improving the underlying Project Merit. A project sitting at 40 because it is fundamentally small **may have less useful growth potential** than an already serious project whose experiments are difficult to reproduce. From there, the optional roadmap can turn that weakness into **3–5 concrete improvements ranked by impact**. Those improvements can be exported as a checklist or **opened as GitHub issues**, which turns the evaluation into actual repository work rather than another dashboard metric. The **Merit Predictor** explores the same idea from a narrower direction. Instead of asking *“How do I improve this entire project?”*, you can choose one rubric dimension and ask what a realistic improvement might look like. The system can estimate a target, describe the work required, and show the resulting **hypothetical** Project Merit. *That final word is important.* A prediction does not change the real score. A generated roadmap does not change the real score either. **Only modifying the project and rescanning new evidence can change what AI-Atlas says the repository demonstrates.** At that point, the evaluator becomes less like a ranking machine and **more like a feedback loop** between what you think you built and what your public repository can actually prove. *AI evaluation sounds simple until you actually try to define what should be evaluated.* A repository contains code, but code alone **does not immediately tell us** whether the project is difficult, original, complete, reproducible, or useful. Those judgments require interpretation. And the moment AI begins interpreting somebody’s work, I think the standard should become **more transparent, not less**. That is the principle I am trying to preserve with **AI-Atlas**. The score should never be the end of the explanation. **Ideally**, you should be able to travel from the overall Atlas score to a domain, from that domain to the projects that contributed to it, from the project to an individual rubric judgment, from the judgment to its reasoning, and eventually toward the files that caused the evaluator to reach that conclusion. **That chain matters more to me than making the number appear authoritative.** That would introduce exactly the kind of false precision the evidence system is supposed to reduce. A much more useful workflow is: *this developer claims NLP & LLM experience; these repositories support that claim; this is what the evaluator found technically significant; and these are the files worth inspecting manually.* The application makes technical work easier to read. **It does not remove the need for human judgment.** **That limitation is important.** An evaluator becomes much less useful the moment it pretends to know more than the available evidence allows. But I still think the underlying problem is worth solving. GitHub **contains one of the richest public records** of technical work ever created, yet we often summarize a profile with **stars, followers, contribution graphs, repository counts**, and **programming languages.** Those are useful signals, but they simply answer different questions. If you have public AI repositories, you can [map your GitHub for free on AI-Atlas](https://getaiatlas.com/?ref=medium2). You do not need an account for the initial scan. Claiming the resulting Atlas lets you inspect the deeper evaluator evidence, join the global, country, and domain leaderboards, use the self-updating GitHub badge and Atlas cards, and work through the project’s improvement tools. The public site currently describes the same core promise in a much shorter sentence: **“Map what you’ve built.”** I am still building AI-Atlas, and difficult repositories are often the best test cases. Every disagreement is another opportunity to make the evaluation more careful, more inspectable, and more grounded in what the code actually demonstrates. I built AI-Atlas because writing *AI Engineer* became easy, but.. **I want demonstrating the engineering behind it to matter more.** **Beyond AI-Atlas**, I regularly write about the systems and research behind modern artificial intelligence — from local inference hardware and GPU memory constraints to model architectures, optimization, multimodal systems, and the mathematics underlying machine learning. My focus is usually the same: **look past the headline and inspect the mechanism**. AI-Atlas applies that philosophy to developer portfolios. Instead of stopping at a title, README, imported library, or GitHub star count, it asks what evidence the implementation itself can support. If you are interested in AI research, practical machine-learning engineering, local infrastructure, and building systems rather than simply using them, you can follow my work here on Medium. Beyond writing about AI infrastructure, I actively build and optimize Data Science workflows. Whether you’re navigating complex datasets, building predictive ML models, or needing end-to-end Data Science assistance for commercial or research projects, **I am available for consulting, and collaborative research.** [I Built an AI to Judge AI Projects — Here’s What It Looks For](https://pub.towardsai.net/i-built-an-ai-to-judge-ai-projects-heres-what-it-looks-for-93b5e1ab4534) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.