Keenable AI Open-Sources NEEDLE: A Live Search Benchmark That Rebuilds Its Query Set Every Hour
MarkTechPost Asif Razzaq
Keenable open-sourced NEEDLE, a live benchmark for search APIs that rebuilds its tests every hour or day. It’s meant to stop agents from peeking at the answer key and to show what search really found.
Based on reporting by MarkTechPost, Asif Razzaq — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Benchmarking web search has gotten awkward in the worst possible way: if the test answers are sitting in public, the thing being tested can simply go fetch them. And if the answer is already buried in a model’s memory, a correct reply doesn’t prove search worked at all. Keenable’s NEEDLE is built around that problem. It is an open-source harness that regenerates its query set from fresh public sources instead of freezing it once and calling that science.
The setup is intentionally live. News queries refresh hourly from RSS feeds and Google Trends. The other verticals — finance, scholar, legal, and rare-entity deep-tail queries — regenerate daily from sources such as SEC XBRL, arXiv, Europe PMC, CourtListener, public agent logs, and related public datasets. Fifteen search APIs are run on the same query text under one protocol, which makes the comparison less about who got the easiest prompt and more about who actually surfaced something useful.
NEEDLE is an acronym: News, Everyday, Expert, Deep-tail, and Legal Evaluation. Each slice is meant to resemble a different kind of agent request. News turns fresh stories from roughly 124 curated RSS feeds and Google Trends into keyword queries. Finance checks registry facts and single-quarter 10-Q numbers. Scholar takes one paper and turns it into four different query styles. Legal pulls recent opinions across 14 federal courts plus eCFR sections. Deep-tail goes after rare words and odd corners of real agent-trajectories.
The scoring is split too. News and deep-tail do not have a single correct result, so an LLM judge rates results from 0 to 4 and the harness reports nDCG@5 with a duplicate-URL penalty. Finance uses answer-recall@5, while scholar and legal are known-item tasks scored by identifier match. Judging uses clipped evidence, not full page fetches, and the judge never sees the engine name. That keeps the evaluation focused on what the engine actually surfaced, not what it might have learned after the fact.
The clever part is the “ultimate” ceiling. NEEDLE pools everything every engine found into a synthetic oracle engine and ranks that combined set by relevance. That gives the benchmark an empirical ceiling for each query — not perfection, just the best the field managed to retrieve. On the published 7-day means ending 2026-08-28, finance is close to solved, while deep-tail remains ugly: Exa leads at 0.557 of ultimate there, with Keenable at 0.470 and Bing far behind. The point is not that one search API is magical. It’s that the gap between retrieval and ranking is now measurable, which is more honest than most leaderboard theater.
My take — AI-written commentary, not fact-checked reporting
This is the right kind of open source: a benchmark that makes cheating harder instead of easier. Search vendors love clean leaderboards; live query regeneration ruins that comfort, which is exactly why it matters. The broader lesson is simple: if a benchmark can be memorized, it eventually becomes a marketing deck, and nobody needs another one of those.
Read more about this at: MarkTechPost