NEEDLE rebuilds search benchmarks before engines can memorize them

Keenable.ai has open-sourced NEEDLE, a live search benchmark that generates fresh queries hourly or daily from public data sources to prevent search engines from memorizing test answers, and it evaluates 15 search APIs across five workloads including news, finance, scholar, legal retrieval, and agentic_rare. The benchmark, available as a Python command-line harness on GitHub, uses dynamic query generation and scoring methods such as nDCG@5 for news and deterministic matching for finance, with no fixed answer files committed to the repository.

• 6 min read NEEDLE rebuilds search benchmarks before engines can memorize them Keenable’s open-source NEEDLE benchmark refreshes search queries hourly or daily and compares 15 APIs against a pooled retrieval ceiling. Image: MarkTechPost https://www.marktechpost.com/2026/08/31/keenable-ai-open-sources-needle-a-live-search-benchmark-that-rebuilds-its-query-set-every-hour/ A search benchmark becomes less useful when the systems being tested can download its answer key. Keenable.ai’s open-source NEEDLE addresses that problem by generating new queries from live public data instead of publishing a permanent test set that search agents can memorize. The benchmark evaluates search APIs against five workloads: news, finance, scholar, legal retrieval and agentic rare , a stream of unusual queries drawn from real agent activity. News queries refresh hourly; the other workloads generate fresh material daily from sources including RSS feeds, Google Trends, SEC XBRL, arXiv, Europe PMC, CourtListener and public search-agent logs. The project is available as a Python command-line harness in the official NEEDLE repository https://github.com/keenableai/needle . It is not a search product. A typical run has two stages: text needle <benchmark generate … --out queries.jsonl needle <benchmark run --queries queries.jsonl … The repository commits none of the generated query or gold-answer data. That separation means a model can inspect the code and scoring rules, but it cannot simply retrieve a fixed answer file during evaluation. What the five benchmarks actually test NEEDLE does not use one generic relevance score for every search task. Each module targets a different failure mode in agentic retrieval. News starts with the newest item from about 124 curated public feeds , usually within a one-hour freshness window, and can also ingest Google Trends RSS across all 52 US geographies . An LLM projects each item into a keyword query and rejects evergreen topics. Results are judged on a 0-to-4 relevance scale and scored with nDCG@5 , including a penalty for duplicate URLs. Finance uses Wikidata and GLEIF for company facts and SEC XBRL for single-quarter 10-Q figures. The registry suite can ask for a CEO, headquarters country, founding year, website, employees, legal entity identifier or ticker. The filings suite defaults to net income, operating income and diluted EPS, with revenue available as an additional field. It scores whether the answer appears in the top K results rather than requiring one particular URL, using deterministic matching with tolerances for names, money values and employee counts. Scholar is a known-item test designed to separate metadata search from full-text retrieval. Each paper produces four query forms: a degraded title, a detail that appears only in the full text, a natural-language clue and a vague “tip-of-the-tongue” description. The system must return the correct arXiv ID, DOI or PMID. A paper remains in the cohort only when it passes the requested bucket checks, keeping comparisons between query types meaningful. Legal retrieval uses recent CourtListener opinions from 14 federal courts and eCFR sections. It checks identifiers such as reporter citations, docket numbers, CourtListener or Justia URLs, and CFR citations. About half of finance and legal queries include operators such as site: , after: or before: ; NEEDLE parses those operators and maps them to each engine’s native mechanism where possible instead of sending unsupported syntax as literal text. The agentic rare module is the closest match to the long-tail queries used by research agents. Its filter keeps English queries containing at least one rare word—defined as a term that BERT WordPiece splits into at least five subwords or maps to UNK . It removes URL-like strings, VINs, hashes, crypto addresses and near-duplicates, then samples queries by short, medium and long length buckets. Like news, it uses an LLM relevance judge and nDCG@5. A pooled ceiling shows whether retrieval or ranking is broken Every engine receives identical query text. The runner makes one request at a time, caps evidence at 2,000 characters per result and evaluates the title and snippet exactly as returned. It does not fetch pages or rerank the providers' output. Search and judging failures are reported separately and excluded from the mean rather than counted as zero. NEEDLE’s ultimate engine pools the deduplicated results from all participating engines for each query and creates an oracle-ranked list: judge ratings determine the order for relevance-scored tasks, while the gold identifier determines the order for known-item retrieval. That gives each individual provider a ceiling based on what the field collectively retrieved. A low score against a high ultimate value indicates a ranking or surfacing problem. A low ultimate value means the combined providers failed to retrieve strong evidence at all. This distinguishes a weak individual engine from a gap shared by the providers. The published seven-day averages below cover the window ending August 28, 2026 . Finance is comparatively strong, while rare-entity retrieval leaves a much larger gap to the pooled ceiling. | Workload | Leading reported result | Other reported results | Ultimate ceiling | |---|---|---|---| | Finance | Exa 0.910 | Keenable 0.872; Perplexity 0.871; Google 0.847 | 0.965 | | Scholar | Keenable 0.774 | Tavily 0.310 | 0.869 | | Agentic rare | Exa 0.557 | Keenable 0.470; Bing 0.199 | Not stated | The agentic-rare figure shows the gap: the leading engine reached only 0.557 of ultimate , and Keenable reached 0.470 . The benchmark’s own documentation describes this as a query-retrieval problem closer to research-agent traffic than title-based lookup. For scholar, the spread reflects the difference between queries answerable from paper metadata and queries whose identifying evidence exists only in the body. Latency also matters because an agent may call search repeatedly during one task. In the same reporting window, Keenable-realtime recorded 193 ms p50 and 284 ms p95 , compared with Exa at 1,876 ms and 2,955 ms , and Bing at 2,767 ms and 9,381 ms . Those figures come from serialized requests under the benchmark protocol, not concurrent load, so they are intended as comparable request latencies rather than a complete measure of system throughput. How to reproduce the runs NEEDLE runs with uv sync and can execute on a laptop or in CI. Most engines require their own API key, while KEENABLE API KEY is optional and falls back to a keyless, rate-limited endpoint. The default model for query projection and judging is openai/gpt-5.6-terra , configurable through NEEDLE LLM MODEL , NEEDLE JUDGE MODEL or command-line overrides. Continuous runs are defined in bench.yaml : news runs hourly with a limit of 20 queries; finance runs at 00:17 UTC with a limit of 120; agentic rare runs at 06:17 UTC with a limit of 100; scholar runs at 12:17 UTC ; and legal runs at 18:17 UTC . Results append summary, URL-overlap and per-engine uniqueness data to GitHub Pages. Full artifacts are archived in the keenable-ai/needle-results Hugging Face dataset, while the repository’s code is released under the MIT license. The benchmark also incorporates findings from the DeepResearchGym research paper https://arxiv.org/abs/2601.17617 , which analyzed 14.44 million search requests across 3.97 million sessions . More than 90% of multi-turn sessions had ten or fewer steps, 89% of inter-step gaps were under one minute, and 54% of newly introduced query terms could be traced to previously retrieved evidence. Those measurements explain why NEEDLE emphasizes fresh query generation, repeated calls and rare terms rather than treating search as a single question-and-answer event. The design has one stated limitation: snippet-only evaluation is a lower bound. A provider may have found the right page but fail to expose the identifying text within its returned snippet, while a known-item score can miss pages that contain no inline identifier. NEEDLE makes those constraints uniform across providers, but it cannot measure information that the API never returns. Frequently asked questions What is the NEEDLE benchmark?+ NEEDLE is an open-source search and retrieval evaluation harness from Keenable.ai. It generates fresh query sets, runs multiple search APIs under one protocol and scores their results. How often does NEEDLE refresh its queries?+ News queries are regenerated hourly. Finance, scholar, legal and agentic rare queries are generated daily from fresh public sources and query streams. Which search engines does NEEDLE test?+ The published comparison covers 15 search APIs, including Keenable, Exa, Bing, Google, Perplexity and Tavily. Each engine requires its own API key when selected. Does NEEDLE evaluate complete web pages?+ No. It evaluates each engine’s returned titles and snippets, with a uniform 2,000-character evidence cap. Pages are not fetched or reranked. Ava Chen /authors/ava-chen/ AI Editor Ava covers the rapidly evolving world of artificial intelligence, from foundational models and research labs to the real-world economics of intelligence. With a background in computational linguistics, she cuts through the hype to find out what actually works. She firmly believes that benchmarks are just marketing until reproduced in the wild.