JioHotstar Benchmarks Knowledge Bases for AI Coding Agents

JioHotstar benchmarked six approaches for helping AI coding agents navigate large codebases and found no single retrieval method consistently improved task completion or reduced token consumption, according to a blog post by Abhishek Bansal, Senior Staff Engineer at JioHotStar. The team compared filesystem tools, Sourcebot, Graphify, CocoIndex Code, Repowise, and an in-house retrieval-augmented generation (RAG) pipeline using subsets of SWE-bench and CodeScaleBench plus a custom question-answering harness, with Claude Sonnet as the solving model, measuring task accuracy, token consumption, agent turns, and tool usability. Sourcebot's indexed text search beat the filesystem baseline on harder SWE-bench Verified tasks at lower cost, while retrieval generally added overhead on simpler SWE-bench Lite tasks and semantic retrieval helped most on CodeScaleBench's ambiguous problem descriptions, suggesting retrieval should be chosen by task characteristics rather than treated as interchangeable.

JioHotstar evaluated six approaches to helping AI coding agents navigate large codebases https://blog.hotstar.com/what-we-learned-from-benchmarking-knowledge-bases-for-coding-agents-6bbdd9e53727 , finding that structural code search performed well on clearly defined engineering tasks, while semantic retrieval helped more with ambiguous requirements. The evaluation also found that adding retrieval layers did not consistently improve task completion or reduce token consumption. Abhishek Bansal https://www.linkedin.com/in/abhishek-bansal/ , Senior Staff Engineer at JioHotStar, compared filesystem tools, Sourcebot https://github.com/sourcebot-dev/sourcebot , Graphify https://github.com/Graphify-Labs/graphify , CocoIndex https://github.com/cocoindex-io/cocoindex-code Code, Repowise https://github.com/repowise-dev/repowise , and an in-house retrieval-augmented generation RAG pipeline. The team evaluated the approaches using subsets of SWE-bench https://www.swebench.com/SWE-bench/ and CodeScaleBench https://github.com/sourcegraph/CodeScaleBench , alongside a custom question-answering harness for JioHotstar's codebase. The experiments used Claude Sonnet as the solving model and measured task accuracy, token consumption, agent turns, and tool usability. JioHotstar’s benchmark setup for evaluating code retrieval approaches for AI coding agents. Source: JioHotstar Blog Post https://blog.hotstar.com/what-we-learned-from-benchmarking-knowledge-bases-for-coding-agents-6bbdd9e53727 The filesystem baseline, which relied on locally checked out repositories and standard search and file operations, remained competitive on concrete tasks. Sourcebot, which provides indexed text search, symbol definitions, references, and file reads, performed better than the baseline on harder SWE-bench Verified tasks while also costing less. On simpler SWE-bench Lite tasks, retrieval generally added overhead without consistently improving results. CodeScaleBench showed a different pattern. Semantic retrieval helped more when agents had to interpret ambiguous problem descriptions and map them to implementation details that were not obvious from exact text or symbol searches. The findings suggest that retrieval approaches should be selected according to task characteristics rather than treated as interchangeable solutions. The report also highlighted the importance of tool design. Agents frequently preferred familiar developer operations such as searching text, reading files, and locating symbols over higher-level abstractions that required additional verification. Tool call breakdown by category across coding agent knowledge base approaches. Source: JioHotstar Blog Post https://blog.hotstar.com/what-we-learned-from-benchmarking-knowledge-bases-for-coding-agents-6bbdd9e53727 Siddharth Vaithinathan https://www.linkedin.com/in/siddharth-vaithinathan-629a8150/ raised a broader architectural question in a LinkedIn post discussing the study https://lnkd.in/p/g2whwQcd : Whether coding agents should understand not only the code but also the company that produced it. He argued that repository indexing alone cannot capture why architectural decisions were made, which constraints shaped them, or what earlier incidents taught the engineering team. Vaithinathan suggested extending agent knowledge beyond code and documentation to include architecture, constraints, decisions, rationale, incidents, and outcomes. Such context could help agents evaluate existing design choices before proposing changes, particularly in systems where historical operational experience influences architectural decisions. Tarun Bhargava https://www.linkedin.com/in/tarun-bhargava-3300954b/ raises a relevant question for benchmark interpretation, particularly when evaluations use limited subsets and probabilistic models. Cost/accuracy numbers with multiple runs of the same setup. JioHotstar's evaluation did not identify a single retrieval approach that performed best across all tasks. Instead, JioHotstar recommends retaining filesystem tools as a baseline, using structural search for concrete, code-focused problems and semantic indexing for ambiguous or cross-cutting requirements. JioHotStar engineering team also emphasizes measuring tool-use traces alongside task accuracy to determine whether retrieval reduces unnecessary exploration. The findings highlight the importance of matching retrieval capabilities to the programming task and how the agent navigates the codebase.