Agentic AI doesn’t have a model problem. It has a retrieval problem.
The New Stack Jenny Morris
GPT-4.1 scored 93.49% with the right docs in front of it, but only 14.58% when an agent had to fetch them. The model wasn’t the problem; the retrieval layer was.
Based on reporting by The New Stack, Jenny Morris — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Agentic AI keeps getting judged on model intelligence, but this piece from The New Stack argues the real bottleneck is getting the right evidence to the model in the first place. The headline example is stark: on BrowseComp-Plus, GPT-4.1 answered 93.49% of questions correctly when the supporting documents were handed to it directly, then dropped to 14.58% when an agent had to find those same docs through BM25 search. Qwen3-32B fell from 83.25% to 3.49% across the same two setups.
That gap is why the benchmark matters. BrowseComp-Plus uses a fixed corpus of 100,195 documents and 830 research questions, each with human-verified supporting documents, so it cleanly separates the model’s reasoning from the retrieval layer’s performance. In the paper’s “oracle” setting, the model gets the labeled positive documents outright. In the agent setting, it has to search. Same model. Very different outcome.
And that is the broader point: production search is messy in ways demos are not. Documents get duplicated. Drafts linger. Teams use different words for the same thing. Several versions of a policy may coexist, and the task can drift beyond the neat shape of an evaluation set. If the agent pulls back stale or wrong material, a smarter model just reasons more confidently about bad evidence.
The article also argues that agentic search behaves differently from human search. People arrive with context; agents often do not. An agent can fan out many searches, combine results, then search again based on what it learned, which means retrieval quality, tail latency, and capacity all start to matter at task scale. Even a simple five-step workflow with 90% success at each step only reaches about 59% end-to-end if nothing can recover.
The interface matters too. A human search box was built for short, underspecified queries. An agent can do much more: exact terms, filters, dates, grouping, ranking choices, follow-up searches. The piece argues for an agent-ready schema that exposes those controls clearly, instead of hiding them behind a generic free-text query or a translation layer that may distort what the agent actually meant.
My take — AI-written commentary, not fact-checked reporting
This is the part the AI industry keeps trying to skip: better models do not magically fix a dumb retrieval stack. If the agent can’t find the right documents, the fancy reasoning is just expensive cosplay. Open models versus closed models is a nice argument for conference stages; in production, the boring retrieval plumbing wins or loses the day.
Read more about this at: The New Stack