{"slug": "discuss-three-ai-investigators-one-codebase-the-hidden-problem-behind-ai-agents", "title": "Discuss+ Three AI Investigators, One Codebase: The Hidden Problem Behind AI Agents", "summary": "A developer released Sentinel-IR-MCP v0.2.2, an MCP-based tool for AI-assisted code investigation, and benchmarked three ways of feeding code to GPT-6 Astra across 87 questions. The full-code approach answered 84 correctly, a compact pre-summarized approach answered 82 with five unresolved, and a hybrid approach that starts from a summary and opens original code on demand answered all 87 while using roughly 71% fewer estimated input tokens than full-code. The developer cautioned the result is one controlled test, not proof that AI agents are error-free.", "body_md": "Thanks for support..\n\nRelease: \"Sentinel-IR-MCP v0.2.2 on GitLab\" ([https://gitlab.com/jakubrojicek11/sentinel-ir-mcp/-/tree/v0.2.2?ref_type=tags](https://gitlab.com/jakubrojicek11/sentinel-ir-mcp/-/tree/v0.2.2?ref_type=tags))\n\nPackage: \"[sentinel-ir-mcp@0.2.2](mailto:sentinel-ir-mcp@0.2.2) on npm\" ([https://www.npmjs.com/package/sentinel-ir-mcp/v/0.2.2](https://www.npmjs.com/package/sentinel-ir-mcp/v/0.2.2))\n\nThe 30-second summary\n\nImagine asking an AI to investigate a software project. Should it read every line of code, receive a short summary, or start with a summary and open the original code only when it needs more evidence?\n\nI tested these three approaches using Sentinel's live benchmark with GPT-6 Astra. Across 87 questions, the full-code approach answered 84 correctly, the compact-information approach answered 82 and left five unresolved, while the hybrid approach answered all 87 correctly.\n\nThe hybrid approach also used approximately 71% fewer estimated input tokens than the full-code approach. This is a promising result from one controlled test, not proof that AI agents are now error-free.\n\n\"Read the original benchmark and inspect the data yourself.\" ([https://dev.to/jackymencz/sentinel-benchmark-livejson-ifl](https://dev.to/jackymencz/sentinel-benchmark-livejson-ifl))\n\nImagine your company hires an investigator to inspect a software project before it goes into production.\n\nThe investigator has a straightforward assignment:\n\nFind out what the program does, what it can access, whether it communicates over a network, whether it writes files, and whether it contains operations that deserve extra attention.\n\nThere is one complication.\n\nThe project contains hundreds or thousands of lines of code. The investigator cannot simply look at the name of the project and know what happens inside it. They need evidence.\n\nNow imagine giving three versions of the same investigator the same assignment.\n\nThe first receives every original document.\n\nThe second receives a structured report prepared in advance.\n\nThe third starts with the report but can return to the original documents when the report does not contain enough information.\n\nWhich one will do the best job?\n\nThat is the central question behind my Sentinel benchmark.\n\nThere is an important distinction here: these are not three independent AI agents arguing with one another or exchanging messages. The benchmark compares three ways of supplying information to the same named model, GPT-6 Astra.\n\nThe difference is not simply how intelligent the investigator is. It is how much relevant evidence the investigator receives, and how the system responds when that evidence is insufficient.\n\nThe first approach is called RAW.\n\nThe model receives the original source code and answers questions about it.\n\nThink of a detective who receives the complete case file, including witness statements, photographs, reports, and every page of supporting evidence.\n\nNothing needs to be summarized before the detective begins.\n\nThis has an obvious advantage. The relevant detail might be buried in an unexpected place, and the original material is available for inspection.\n\nBut there is a cost.\n\nEvery time the model needs to investigate a question, the supplied information can contain much more material than that question actually requires.\n\nSuppose you ask whether a program exposes an HTTP endpoint. The answer might be visible in a handful of lines. Yet the model may also receive code responsible for unrelated calculations, formatting, logging, and other operations.\n\nThe extra material is not necessarily harmful. It is simply not always relevant to the current question.\n\nIn AI systems, the pieces of information supplied to a model are often represented as tokens. Tokens are the units models process when reading text and producing responses.\n\nA larger input can require more processing and may increase usage costs, depending on the model and service.\n\nThe important point is that giving an AI more information is not automatically the same as giving it better information.\n\nSometimes, the extra pages contain exactly what the investigator needs. Sometimes, most of them are irrelevant to the question.\n\nThe second approach is called IR, short for Intermediate Representation.\n\nInstead of receiving only the original code, the model receives a structured description of important facts extracted from that code.\n\nImagine receiving a building's floor plan rather than having to inspect every room before answering a basic question.\n\nThe plan might identify the entrances, exits, electrical rooms, and locations of restricted areas. You can answer many questions from that map without exploring the entire building.\n\nSentinel-IR applies a similar idea to software.\n\nIts output can identify facts such as:\n\nThis is more than asking an AI to write a shorter version of a long file.\n\nA normal summary might say, \"This program runs a server and processes requests.\"\n\nA structured representation can instead record individual routes, imported modules, operations, and other specific facts in a predictable format.\n\nThe distinction matters because the AI is no longer expected to rediscover every structural detail from the original text whenever it needs to answer a question.\n\nIt receives a prepared map.\n\nThere is a catch, however.\n\nA map is only useful if it accurately records the important features of the territory.\n\nIf a relevant operation is missing from the representation, the model may never see the evidence it needs. And if the representation simplifies something too aggressively, the model might reach the wrong conclusion.\n\nA compact representation is therefore not automatically a better representation.\n\nIt must preserve the information that matters.\n\nThis is where the story becomes interesting.\n\nPeople often assume that a compressed or summarized version of something must always be smaller than the original.\n\nThat is not true for structured software information.\n\nIn one small example in the published benchmark, a 27-line HTTP server had an estimated input size of 163 tokens for the original source and 272 tokens for its IR representation.\n\nThe structured version was larger.\n\nWhy?\n\nBecause the original file was already relatively small. A structured report also needs labels, field names, lists, and descriptions to make its facts understandable to the model.\n\nThose additional elements can cost more than the original text when the source is short.\n\nNow consider a much larger file.\n\nFor a sample of approximately 393 lines, the benchmark records an estimated 2,861 tokens for the source and 734 for the IR representation.\n\nFor a sample of approximately 951 lines, the figures are 7,186 tokens for the source and 498 for IR.\n\nSuddenly, the map becomes much smaller than the territory.\n\nThat is the practical reason to investigate this approach.\n\nIt is not a promise that every file will shrink. It is an attempt to avoid repeatedly sending large amounts of source code when a compact set of relevant facts can answer the question.\n\nThe larger the difference between the original material and the useful information required for a task, the more opportunity there may be to save input tokens.\n\nThe third approach is called IR + RAW.\n\nIt combines the two previous approaches.\n\nThe model starts with structured information. When that information is not sufficient, the workflow can escalate and consult the original source.\n\nReturn to our building inspector.\n\nThe inspector begins with a floor plan. It shows the main entrances, exits, and restricted areas.\n\nMost questions are easy to answer from the plan.\n\nBut then the inspector notices a door leading into a room marked as restricted. The plan identifies the door, but it does not explain what happens inside the room.\n\nA sensible inspector does not guess.\n\nThey request access to the original evidence.\n\nThat is the idea behind the hybrid approach: use compact information where it is enough, and retain a path back to the original source when more detail is required.\n\nThis is different from choosing between two extremes.\n\nThe system does not have to send everything every time, and it does not have to pretend that its initial summary contains every possible answer.\n\nIn the benchmark, the IR + RAW variant recorded 93 model calls for 87 questions. That means six additional calls beyond the one-call-per-question baseline, consistent with targeted escalation in this test.\n\nThis additional work matters. Access to the original source is not free, and extra calls can add time.\n\nBut neither is a wrong answer free. In a software security workflow, an incorrect conclusion about a dangerous operation can be far more expensive than an additional investigation.\n\nThe real design question is therefore not, \"How can we eliminate every extra call?\"\n\nIt is, \"When is the extra call worth making?\"\n\nThe published Sentinel benchmark records a live run using GPT-6 Astra across 87 questions.\n\nHere are the reported results.\n\nApproach| Correct answers| Unresolved| Estimated input tokens\n\nRAW| 84 of 87| 0| 279,476\n\nIR| 82 of 87| 5| 58,549\n\nIR + RAW| 87 of 87| 0| 80,340\n\nThe accuracy figures reported by the benchmark are 96.6% for RAW, 94.3% for IR, and 100% for IR + RAW.\n\nThe differences tell us something important.\n\nFirst, the compact representation substantially reduced the measured input volume.\n\nIR used approximately 79% fewer estimated input tokens than RAW.\n\nSecond, reduced input volume did not automatically preserve the full result.\n\nThe standalone IR approach left five questions unresolved. It used less input, but it did not answer every question.\n\nThird, the hybrid approach recovered the missing answers in this particular run.\n\nIR + RAW used approximately 71% fewer estimated input tokens than RAW and answered all 87 questions correctly according to the benchmark's expected answers.\n\nThe results support further investigation into a system that combines compact facts with access to original evidence.\n\nThey do not establish that the hybrid approach will always outperform the alternatives, or that it is ready to make unsupervised production decisions.\n\nEighty-seven questions are useful evidence, not a guarantee about every possible codebase.\n\nThere is a technical limitation that matters when interpreting these results.\n\nThe benchmark estimates token counts using the length of the text divided by four. It applies that approximation consistently across the variants.\n\nThese figures are therefore useful for comparing the approximate size of their inputs. They are not exact measurements of the model's actual tokenization or the money charged by an AI provider.\n\nThe distinction matters because real token counts vary with the content and tokenizer, and costs also depend on the model and the service's pricing.\n\nThe elapsed times in this run reinforce another point: fewer estimated input tokens did not automatically make the hybrid workflow the fastest.\n\nThe recorded elapsed times were approximately 155 seconds for RAW, 147 seconds for IR, and 159 seconds for IR + RAW. These are measurements from this particular run, not general performance guarantees.\n\nSo the strongest result here is not that the hybrid system is always cheaper, faster, or more accurate.\n\nIt is that this hybrid configuration answered all the tested questions correctly while using substantially fewer estimated input tokens than the full-source approach.\n\nThat is a specific, testable claim.\n\nIt is also a much more useful claim than simply announcing that AI has become smarter.\n\nNow imagine applying the same system to a real software project.\n\nA program might read an API key, communicate over a network, write files, or launch another process. These operations can be legitimate. Their presence alone does not prove that a program is malicious.\n\nBut they may matter when assessing what the program can do and what could go wrong.\n\nAn AI system investigating the project needs to distinguish between several things:\n\nWhat does the code contain?\n\nWhat does the available evidence allow us to conclude?\n\nWhat information is still missing?\n\nAnd when should the system refuse to make a confident claim?\n\nThese are separate questions.\n\nA report identifying a process-launching operation is evidence about the program's structure. It is not, by itself, a complete security verdict.\n\nLikewise, a missing finding does not prove that an operation is absent unless the analysis method is reliable enough to support that conclusion.\n\nThis is why compact representations need more than clever compression. They need accurate extraction, carefully defined facts, and a way to handle missing information.\n\nIt is also why evaluation matters.\n\nA result that looks impressive in a demonstration must be tested against known answers, difficult cases, and situations where the system should admit that it does not know enough.\n\nThe benchmark is a small example of testing this design question with concrete questions and recorded answers rather than relying only on a persuasive demonstration.\n\nThe bigger idea extends beyond source code.\n\nAI agents increasingly work with documents, databases, websites, tools, and information accumulated over multiple steps.\n\nIn all these settings, the system must decide what information to keep available, what to retrieve, and what to do when the evidence is incomplete.\n\nImagine a customer-support agent investigating a delayed order. It may not need the customer's entire history. It needs the order number, shipping status, relevant updates, and perhaps access to the original records when a discrepancy appears.\n\nImagine a research agent. It may start with notes and extracted findings, then reopen the underlying paper to verify an important detail.\n\nImagine an AI coding assistant. It may need a compact map of a repository before making a change, but it should still inspect the actual implementation before modifying a critical function.\n\nThese are examples of the same design principle, not results demonstrated by this particular benchmark.\n\nThere is also a broader lesson about multiple agents.\n\nAdding more AI agents does not automatically make a system better. Agents need useful information, clear responsibilities, reliable communication, and a way to detect mistakes. Otherwise, a larger system can simply create more opportunities for confusion.\n\nSometimes the answer is to add another agent. Sometimes it is to give the existing agent better evidence. Sometimes it is to use a deterministic tool instead of asking a model to infer something that can be checked directly.\n\nThe right design depends on the task.\n\nFor Sentinel, the question is narrower: can a structured representation provide enough information for many software-inspection questions, while preserving a route back to the original code when it does not?\n\nThe live benchmark provides an encouraging result for that approach. The next step is to repeat the experiment across more projects, more questions, different kinds of code, and independent runs, while measuring actual token usage and costs where possible.\n\nThat is how a promising result becomes stronger engineering evidence.\n\nLet's return to our investigator.\n\nGiving the investigator every document can work, but it may be wasteful.\n\nGiving them only a summary can be efficient, but it may leave important questions unanswered.\n\nGiving them a structured map and the ability to inspect original evidence offers another possibility.\n\nThe investigator can use the map for ordinary questions and return to the evidence when necessary.\n\nThat is the idea behind Sentinel's IR + RAW approach.\n\nThe goal is not to make an AI appear clever by giving it less information. The goal is to give it the right information for the task, recognize when that information is insufficient, and make further investigation possible.\n\nThe 87-question benchmark does not prove that this approach is universally superior. It does show why the approach deserves further testing.\n\nAnd it raises a question that matters well beyond software security:\n\nWhen we build AI agents, should we focus only on making them think better, or should we also improve how they receive evidence, recognize uncertainty, and decide when to investigate further?\n\nI believe the second part deserves far more attention.\n\nBecause an intelligent investigator with incomplete evidence can still reach the wrong conclusion. A well-designed investigative process gives that investigator a chance to discover what is missing.\n\nTen articles and research papers worth reading next\n\nThe Sentinel benchmark explores one specific problem: how an AI system uses information. These ten resources examine related challenges, from context selection and multi-agent coordination to security and evaluation.\n\n\"Effective Context Engineering for AI Agents — Anthropic\" ([https://www.anthropic.com/research/effective-context-engineering-for-ai-agents](https://www.anthropic.com/research/effective-context-engineering-for-ai-agents))\n\nWhy the information supplied to a model needs to be managed throughout an agent's work, not just written into a good initial prompt.\n\n\"Towards a Science of Scaling Agent Systems — Google Research\" ([https://research.google/blog/towards-a-science-of-scaling-agent-systems-when-and-why-agent-systems-work/](https://research.google/blog/towards-a-science-of-scaling-agent-systems-when-and-why-agent-systems-work/))\n\nResearch on when multiple agents improve performance and when coordination can make a system worse.\n\n\"A Practical Guide to Building AI Agents — OpenAI\" ([https://openai.com/business/guides-and-resources/a-practical-guide-to-building-ai-agents/](https://openai.com/business/guides-and-resources/a-practical-guide-to-building-ai-agents/))\n\nA practical introduction to agent architecture, tools, workflows, and safeguards.\n\n\"ContextBench: A Benchmark for Context Retrieval in Coding Agents\" ([https://arxiv.org/abs/2602.05892](https://arxiv.org/abs/2602.05892))\n\nResearch examining whether coding agents retrieve and use the relevant parts of a codebase when solving problems.\n\n\"SWE Context Bench: A Benchmark for Context Learning in Coding\" ([https://arxiv.org/abs/2602.08316](https://arxiv.org/abs/2602.08316))\n\nResearch on reusing previous experience and summaries across related coding tasks, including the risks of retrieving the wrong information.\n\n\"AgentBench: Evaluating LLMs as Agents\" ([https://proceedings.iclr.cc/paper_files/paper/2024/hash/e9df36b21ff4ee211a8b71ee8b7e9f57-Abstract-Conference.html](https://proceedings.iclr.cc/paper_files/paper/2024/hash/e9df36b21ff4ee211a8b71ee8b7e9f57-Abstract-Conference.html))\n\nA benchmark evaluating how language models reason, follow instructions, and act across interactive environments.\n\n\"Systematic Debugging for AI Agents: Introducing AgentRx — Microsoft Research\" ([https://www.microsoft.com/en-us/research/blog/systematic-debugging-for-ai-agents-introducing-the-agentrx-framework/](https://www.microsoft.com/en-us/research/blog/systematic-debugging-for-ai-agents-introducing-the-agentrx-framework/))\n\nHow to identify where a long AI-agent workflow failed instead of looking only at its final answer.\n\n\"Know Your Limits: A Survey of Abstention in Large Language Models\" ([https://aclanthology.org/2025.tacl-1.26/](https://aclanthology.org/2025.tacl-1.26/))\n\nResearch on when an AI should refrain from answering because it cannot provide a sufficiently reliable response.\n\n\"OWASP Top 10 for Agentic Applications 2026\" ([https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/))\n\nSecurity risks associated with AI systems that can plan, use tools, and take actions.\n\n\"Evaluate Agent Workflows — OpenAI\" ([https://developers.openai.com/api/docs/guides/agent-evals](https://developers.openai.com/api/docs/guides/agent-evals))\n\nPractical guidance on evaluating agent runs, examining failures, and testing whether changes actually improve a workflow.\n\nThe numbers in this article come from the published Sentinel live benchmark. The interpretation of the results is deliberately limited to that experiment; broader performance, security, and cost claims require further testing.\n\nOriginal benchmark: [https://dev.to/jackymencz/sentinel-benchmark-livejson-ifl](https://dev.to/jackymencz/sentinel-benchmark-livejson-ifl)", "url": "https://wpnews.pro/news/discuss-three-ai-investigators-one-codebase-the-hidden-problem-behind-ai-agents", "canonical_source": "https://dev.to/jackymencz/discuss-three-ai-investigators-one-codebase-the-hidden-problem-behind-ai-agents-2cf1", "published_at": "2026-10-24 16:40:03+00:00", "updated_at": "2026-10-24 16:46:42.143550+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "large-language-models", "ai-research"], "entities": ["Sentinel-IR-MCP", "GPT-6 Astra", "GitLab", "npm", "Sentinel"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/discuss-three-ai-investigators-one-codebase-the-hidden-problem-behind-ai-agents", "markdown": "https://wpnews.pro/news/discuss-three-ai-investigators-one-codebase-the-hidden-problem-behind-ai-agents.md", "text": "https://wpnews.pro/news/discuss-three-ai-investigators-one-codebase-the-hidden-problem-behind-ai-agents.txt", "jsonld": "https://wpnews.pro/news/discuss-three-ai-investigators-one-codebase-the-hidden-problem-behind-ai-agents.jsonld"}}