Performance engineering from kernel analysis to AI: Adrian Cockcroft’s take

At P99 CONF, Adrian Cockcroft didn’t come right out and call P99 a bad fit for modern systems. He didn’t need to. The long-time performance engineer used his talk to make a sharper point: a single percentile can hide the shape of what’s really happening inside a service, especially when latency isn’t one smooth line but a mix of distinct modes.

Cockcroft’s instinct is still the same one he had at Sun Microsystems decades ago. When the old tools showed him vmstat output and not much else, he went to the kernel source, traced where the numbers came from, and wrote it all down. That work led to two books, Sun Performance and Tuning and Resource Management. Even then, the goal was not just to read metrics, but to understand what the tools were failing to show.

Now he’s doing the same thing with a different set of helpers. Instead of spending time setting up custom scripts and chasing code fragments, Cockcroft says he can use AI to build analysis tools quickly. He described that speedup in blunt terms: the code would not have existed otherwise. The latest example is open-source tooling he built to study response-time distributions rather than flattening everything into averages and percentiles.

That distinction matters because a single histogram can contain more than one peak. Cockcroft used the cache-hit-versus-cache-miss case as the clean example: the fast path and the slow path stay in place, but their relative heights shift as hit rates change. The average moves. P99 moves. The real system behaviour, though, is just that the mix changed. His tool, written in R with help from ChatGPT, identifies an arbitrary number of peaks and follows how they change over time.

His advice to teams was simple: start wide, then zoom in. Look at the macro view first, then keep digging until the slow request is in focus all the way down. That is classic Cockcroft, really — less worship of dashboards, more suspicion of them.