It passed CI. It passed your evals. The customer still got the wrong answer.

A diff is not proof. It only says what changed. In AI systems, that’s often not enough. The change ships, the checks pass, and then the customer gets a slow reply or a wrong one, while the logs still look calm. That is the gap Dynatrace is pointing at in its 2026 State of SRE and Platform Engineering report, based on 919 enterprise leaders around the world: 77% of platform engineering teams have observability in at least some services, but only 40% have it fully wired into every deployment.

That split used to be annoying. With ordinary services, failures usually shout. A 500. A timeout. A dead dependency. AI agents can fail in a much quieter way. They can return a 200, pass faithfulness checks, and still hand the user the wrong answer. So the trick is not to stare at the score and feel reassured. It is to find the request, then reconstruct what the system actually did.

The article’s example is a support agent that answers questions from product docs. After a coding assistant rewrites the lookup path, CI and evals both pass. Then answers slow down, and some cite older versions. The useful clues are in the trace: release, retrieval config, feature-flag state, the repeated searches, and the fact that the version filter is null. That pattern says something important. The model may not be the thing at fault. The harness, the retries, or the retrieval layer may be.

And that is why the piece keeps coming back to evidence from the running system. One trace shows three searches, another shows a model call between each search, and those are different bugs. A grounded answer can still be wrong if the retrieved documents were valid for the query but invalid for the requested version. The faithfulness score won’t catch that. A retrieval judge might not either. The missing assertion is simpler: did the requested version actually reach the lookup?

The practical advice is blunt. Put the release, retrieval settings, document IDs, and versions on the trace. Keep the evidence linked, not scribbled on a closed span. Test the lookup directly in CI, then judge the answer separately. And when the numbers move, watch latency, tool calls, and token counts together, because each one can lie in a different direction.