{"slug": "inside-hackerrank-s-llm-based-hiring-agent", "title": "Inside HackerRank's LLM-based Hiring Agent", "summary": "HackerRank open-sourced its LLM-based Hiring Agent, a resume scoring tool that parses PDFs, enriches data with GitHub and blog content, and returns a structured score. The tool uses prompts for evaluation, with scoring criteria that may encode biases, and relies on Markdown conversion that can be unreliable for multi-column layouts.", "body_md": "I have been doing my goal-setting, KPIs, and feedback assessments through LLMs. The hardest part was designing the scoring mechanism. When HackerRank open-sourced their [Hiring Agent](https://github.com/interviewstreet/hiring-agent), I wanted to understand how they had solved the same problem — and what biases their rubric encodes.\n\n[Hiring Agent](https://github.com/interviewstreet/hiring-agent) is an LLM-based resume scoring tool that parses a PDF, enriches it with GitHub and blog data, and returns a score. It’s straightforward to run — clone the repo, point it at a PDF:\n\n``` bash\n$ git clone https://github.com/interviewstreet/hiring-agent\n$ cd hiring-agent\n$ python3 score.py /examples/java-engineer.pdf\n```\n\nThe tool returns a structured score:\n\n```\n================================================================================\n📊 RESUME EVALUATION RESULTS FOR: Senior Java Engineer\n================================================================================\n\n🎯 OVERALL SCORE: 71.0/100\n🌐 Open Source:          12/35\n🚀 Self Projects:        18/30\n🏢 Production Experience: 25/25\n💻 Technical Skills:      8/10\n\n✅ KEY STRENGTHS:\n------------------------------\n  1. Microservices Architecture\n  2. Event-Driven Systems\n  3. Cloud Technologies\n  4. Database Optimization\n  5. API Design\n\n🔧 AREAS FOR IMPROVEMENT:\n------------------------------\n  1. Lack of Open Source Contributions\n  2. Limited GitHub Activity\n  3. No Publicly Available Projects\n```\n\nI first provided it with a Senior Java Engineer resume. Though the profile has a near-perfect score on production experience and technical skills, the tool still evaluated the candidate at 71. I wanted to get a deeper understanding of how this tool was doing the scoring:\n\n## 1. Resume Parsing\n\nIt starts in `score.py`\n\nwhere the PDF is handed off to a `PDFHandler`\n\n:\n\n**score.py:251-252**\n\n```\npdf_handler = PDFHandler()\nresume_data = pdf_handler.extract_json_from_pdf(pdf_path)\n```\n\nInside extract_json_from_pdf, the PDF is parsed using PyMuPDF and converted to Markdown:\n\n**pdf.py:54-57**\n\n```\nresume_text = to_markdown(\n    doc,\n    pages=pages,\n)\n```\n\nThe Markdown content is then normalised into structured JSON - an internal schema (basics, work, education, skills, projects, awards) that represents the content of the resume. The normalisation step is important as it allows the tool to take in any resume regardless of the layout and represent it into a strict internal format. Now, parsing the resume from PDF to Markdown is not a foolproof step, in case of multi-column sections direct Markdown conversion is unreliable.\n\nLet’s take an example of this resume:\n\nThe Markdown parser reads left-to-right, so the content is incorrectly extracted as:\n\n**Coding, Framework, Messaging, Database, Observability, Containers, Patterns, Cloud, Java, Python…**\n\nHow well the LLM recovers from this depends on the model used. A more concrete solution would be to screenshot the resume and let the model parse it directly, rather than converting to Markdown first - similar to how Claude/Codex parses websites. The sections are then extracted individually using specialized prompts, each being an LLM call to the model.\n\n## 2. Scoring Criteria\n\nAlmost all the intelligence is prompts, available in `prompts/templates/`\n\ndirectory. Python is merely used in orchestration and plumbing.\n\n```\n\"basics\": \"basics.jinja\",\n\"work\": \"work.jinja\",\n\"education\": \"education.jinja\",\n\"skills\": \"skills.jinja\",\n\"projects\": \"projects.jinja\",\n\"awards\": \"awards.jinja\",\n\"system_message\": \"system_message.jinja\",\n\"github_project_selection\": \"github_project_selection.jinja\",\n\"resume_evaluation_criteria\": \"resume_evaluation_criteria.jinja\",\n\"resume_evaluation_system_message\": \"resume_evaluation_system_message.jinja\",\n```\n\n`resume_evaluation_system_message.jinja`\n\nand `resume_evaluation_criteria`\n\nhelp evaluate the scoring. The other prompts are primarily for resume parsing capability. The scoring criteria in `resume_evaluation_criteria`\n\nis heavily biased towards individuals with open-source projects and contributions:\n\n```\n### Open Source (0-35 points)\n**HIGH SCORES (25-35 points):**\n- Contributions to popular open source projects (1000+ stars)\n- Significant contributions to well-known projects\n- Google Summer of Code (GSoC) participation\n- Substantial community involvement\n```\n\nThere are explicit rules that mention what projects are considered open-source and what are not.\n\n```\n**CRITICAL RULES:**\n- Having personal GitHub repositories does NOT constitute open source contribution\n- True open source contribution means contributing to OTHER people's projects\n- When GitHub data shows all projects are 'self_project' type, open source score MUST be 10 points or less\n```\n\nAnd one that interested me most was the following rule:\n\n```\n**SPECIAL CONSIDERATION FOR STARTUP EXPERIENCE**: Give extra points for founder roles, co-founder positions, or early-stage engineer roles (first 10-20 employees) at startups, as these demonstrate exceptional initiative, technical leadership, and ability to build products from scratch.\n```\n\n### Testing the Bias\n\nThe prompt says to reward founder and early-stage roles. I wanted to know: does it actually? I took the same resume and changed only the job titles. Three versions:\n\n[Senior Java Engineer](/assets/pdfs/java-engineer.pdf)[Founding Engineer](/assets/pdfs/founding-engineer.pdf)- same resume, title swapped[Co-founder / CTO](/assets/pdfs/co-founder.pdf)- same resume, title swapped again\n\nEverything else stayed identical: same skills, same projects, same experience. The only variable was the job title.\n\nThe results were underwhelming. The production score moved from 22 to 23 out of 25 in some iterations. The CTO title landed in the same range. I ran it multiple times with different titles, and the production score consistently sat between 20-23 regardless of what I put.\n\nThis tells two things. First, the LLM is resistant to title changes. Second, the rubric’s emphasis on startup roles is more aspiration than enforcement. The prompt says to give extra points, but the model doesn’t consistently apply them. A prompt rule is a suggestion, not a guarantee.\n\nA deeper look into the criteria also rewarded the following:\n\n```\n- +3-5 points for startup founder/co-founder experience\n- +2-3 points for early-stage engineer experience (first 10-20 employees at a startup)\n- +2 points for portfolio website (GitHub URL in basics.url)\n- +1 point for LinkedIn profile\n```\n\nBeing a LinkedIn user is worth 1 point. Being a GSoC participant is worth 5. Now, apart from prompts there are rules explicitly coded wherein it checks for repo popularity:\n\n**github.py:234**\n\n```\nfor repo in repos_data:\n    if repo.get(\"fork\") and repo.get(\"forks_count\", 0) < 5:\n        continue\n```\n\nDrops forks with fewer than 5 forks, filtering out drive-by forks of popular repos. These are opinions stated as rules.\n\n## What I Learned\n\nThe Hiring Agent is a decently-engineered prototype. The problem is someone decided open source is 35% of an engineer’s value; four commits is the threshold for “meaningful” contribution. Someone decided a LinkedIn profile is worth exactly 1 point. These are judgment stated as rules.\n\nThis is the same problem I was trying to solve in my own tool. The hard part of LLM-based scoring isn’t the model, the parsing, or the pipeline. It’s deciding what matters. Once you define it in the prompt, you have made a value judgment and the model will try to enforce it.\n\n## A Caveat\n\nThe git history is also interesting. The active development spans 77 days (July 29 to October 15, 2025) followed by a 231-day silence, then a handful of cleanup commits in June 2026 before open-sourcing. The commit patterns look like a burst of internal hackathon or intern work, not a sustained effort.\n\nThis matters because the company that is open-sourcing the project is one of the players in the domain. The criteria defined: open source at 35%, technical skills at 10% feels like someone’s first draft and not a thoroughly validated hiring framework. There are inconsistencies also in the project that highlight that these are the rough edges of a prototype, not a production system. This tool must be used with caution as it’s a first attempt at LLM-based hiring looks like, before it’s been tested against thousands of real candidates and calibrated by recruiting teams.", "url": "https://wpnews.pro/news/inside-hackerrank-s-llm-based-hiring-agent", "canonical_source": "https://blog.grandimam.com/posts/how-hacker-rank-scores-engineers/", "published_at": "2026-07-11 07:43:55+00:00", "updated_at": "2026-07-11 08:05:06.588556+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-ethics"], "entities": ["HackerRank", "Hiring Agent", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/inside-hackerrank-s-llm-based-hiring-agent", "markdown": "https://wpnews.pro/news/inside-hackerrank-s-llm-based-hiring-agent.md", "text": "https://wpnews.pro/news/inside-hackerrank-s-llm-based-hiring-agent.txt", "jsonld": "https://wpnews.pro/news/inside-hackerrank-s-llm-based-hiring-agent.jsonld"}}