Why basic RAG fails at multi-hop reasoning (and how GraphRAG fixes it)
The New Stack Emmanuel Akita
Basic RAG chokes on questions that need more than one hop. GraphRAG adds relationships, so the model can follow the trail instead of guessing.
Based on reporting by The New Stack, Emmanuel Akita — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
A lot of RAG advice sounds clean on paper: split PDFs into chunks, embed them, throw them into a vector database, and call it done. The problem shows up fast once the system meets real users. Simple similarity search is good at finding text that looks related. It is much worse at answering questions that require connecting separate facts across different parts of the data.
That’s the core failure here. If someone asks who leads the company Acme Corp acquired, a normal vector retriever can easily grab the acquisition statement and miss the leadership statement because those pieces do not live in the same chunk and do not look similar enough to land together. The same weakness shows up in global summaries, where the system has to pull together risk factors scattered across many compliance reports. Chunking text helps storage. It does not create reasoning.
GraphRAG is the proposed fix, and the difference is structural. During ingestion, the LLM extracts nodes and edges from the documents, then stores them as a knowledge graph. At query time, the system uses vector search to find a starting point and then traverses the graph to collect connected context. The article points to Neo4j’s neo4j-graphrag as a way to build that workflow in Python.
The example is straightforward. One chunk says Acme Corp acquired BetaTech in 2022. Another says Sarah Connor was appointed CEO of BetaTech in 2023. A basic vector search might not connect those dots. A GraphRAG setup, with an explicit schema and relationship types like ACQUIRED and LEADS, can pull BetaTech into view and follow the path to the right person.
But this is not free magic. The article is blunt about the trade-offs: ingestion is expensive because the LLM has to extract entities and relationships, and rate limiting matters because the pipeline can hit a RateLimitError if it is not handled properly. Schema design also matters, because without it the graph can fill up with duplicate or messy entity labels. In return, debugging gets easier, since a Neo4j dashboard shows the nodes and relationships instead of a pile of floating-point vectors.
My take — AI-written commentary, not fact-checked reporting
This is the right kind of skepticism about RAG hype: vector search is useful, just not holy. The industry keeps trying to sell one retrieval trick as an answer to every enterprise problem, and that’s how you end up with elegant demos and embarrassing production systems. Graphs are less glamorous, which is probably why they work better.
Read more about this at: The New Stack