Evaluating RAG with LLM as a Judge

Mistral AI has laid out a practical framework for a problem that's been quietly annoying anyone building retrieval-augmented generation systems: how do you actually know if the thing is working? Not "does it produce fluent sentences" — LLMs are good at that regardless of whether they're right. The real question is whether the retrieved documents were relevant, whether the answer sticks to those documents, and whether the answer actually addresses what the user asked. Three separate failure points, and traditional metrics tend to blur them into one fuzzy impression.

The approach Mistral is promoting, called the RAG Triad, isn't new — it originated with TruLens, and similar frameworks like RAGAS have chased the same goal. It splits evaluation into three distinct checks: context relevance (did the retrieval step pull documents that actually match the query), groundedness (does the generated answer stay faithful to those documents, rather than drifting into hallucination), and answer relevance (does the final response actually answer what the user asked). Score each separately and you get a much clearer picture of where a RAG pipeline is breaking, rather than a single vague number that tells you nothing about which layer failed.

To run these checks at scale, without an army of human reviewers, Mistral leans on the increasingly common "LLM as a Judge" pattern: one model generates answers, a second model grades them against a defined scale, whether that's numeric, binary, or a qualitative label like "Good" or "Bad." The catch with judge LLMs has always been consistency — get the same model to grade a hundred similar answers and the outputs can wobble in format and rigor. Mistral's fix is its structured outputs feature, which forces the judge model to return machine-readable, schema-defined results instead of loose prose.

In the cookbook example Mistral shared, that means Pydantic classes for ContextRelevance, AnswerRelevance, and Groundedness, each requiring a step-by-step explanation plus a score. It's a small technical detail, but it matters: reusable schemas mean evaluation criteria can be defined once and applied consistently across thousands of queries, and the structured format makes the results easy to aggregate, chart, or feed into a dashboard without manual parsing. Mistral has published the full notebook on GitHub, so teams can plug this straight into their own RAG pipelines rather than build the scaffolding from scratch.