Debugging our AI search assistant with agent tracing

Sentry’s team built a search assistant to turn natural-language prompts into Sentry Search Syntax, because plenty of users would rather type what they mean than learn a query language first. That sounds simple until the output stops being deterministic. So they leaned on evals: write scenarios, run them, see whether the agent produces the right query, then chase down the misses.

One miss turned into a useful example. During end-to-end testing, the assistant failed on a custom numerical attribute. A shopping app might use a field like that to count items in a cart, and a user might want to find purchase events where the number is greater than 10. Instead, the assistant produced a query that matched nothing.

The useful part was not just that Sentry had logs for the AI flow. It was that the team could debug it the same way they would any other backend issue. They opened the Agents view, filtered by time range, agent type, and their own email, and then walked the trace. The timeline showed the system prompt, the model output, and the tool calls that led to the bad answer. One hop deeper, the trace revealed the backend API call behind the tool result.

That’s where the bug surfaced. The prompt was exposing the available field as bug__predictions_count, but Sentry Search Syntax wanted it in the tagged form tags[bug_predictions_count, number]. The model was being handed the wrong shape of information, so the query came out wrong. Once they fixed that, they added a new eval to cover the case and watched it pass locally.

They also checked the deployed version in production. The same query now returned the right data, and the trace showed the corrected prompt plus the right API parameter. The company liked the pattern enough to send local eval runs to Sentry too, so developers can see traces before opening a pull request. That makes the AI work feel less like mystical slot-machine tuning and more like regular engineering, which is exactly the point.