NVIDIA Nsight Python Tutorial: How to Profile GPU Kernels

Nsight Python is NVIDIA’s open-source Python layer over Nsight Compute, and this tutorial uses it the way programmers actually work now: inside a notebook. Instead of launching a script from the command line and combing through a report later, you mark the code you care about with nsight.annotate, wrap a benchmark with nsight.analyze.kernel, and get a pandas DataFrame back with hardware metrics for each annotated region, configuration, and run.

The setup itself is part of the lesson. On free Colab, the notebook checks the GPU, installs a current Nsight Compute next to the older one Colab already ships, and points PATH at the new launcher. It then installs Nsight Python from the commit the notebook was written against, plus cupti-python in a separate directory because the CUPTI backend needs a newer library than Colab’s PyTorch setup wants.

Order matters here. import nsight has to happen before anything else touches CUDA, because Nsight Compute injects into the running Python process and can only observe work started after that. Once it’s in place, a first fp32 matmul shows the basic flow: only the code inside nsight.annotate is measured, the DataFrame exposes the kernel name, timing, run count, clock fields, and stability flag, and the notebook turns the measured time into TFLOP/s by hand.

From there the tutorial gets more practical. It runs each configuration 20 times to show how much noise matters, and the DataFrame starts carrying standard deviation, confidence intervals, relative standard deviation, and a StableMeasurement flag. Short kernels are twitchy. The source is blunt about that: a tiny vector add can vary a lot, so you judge it with more runs and compare confidence intervals instead of trusting a single average.

Then it shows how profiler defaults change the answer. cache_control decides whether caches are flushed before each replay, and clock_control decides whether the GPU is locked to base clocks or allowed to boost. On the T4 used in the notebook, that matters because the L2 cache is 4 MB, and the reported ComputeClock is just the device’s rated clock attribute, not what the kernel actually ran at. The real run clock comes from gpc__cycles_elapsed.avg.per_second.

The later sections keep pushing on the same idea: use hardware counters, not vibes. The notebook sweeps matrix sizes in fp32 and fp16, derives TFLOP/s, uses counters to separate compute-bound from memory-bound work, measures kernel fusion in bytes moved, records clocks and temperature per run, and compares the experimental CUPTI backend with Nsight Compute itself. It also keeps the raw report around for deeper analysis, which is the right instinct. Nice charts are fine; evidence is better.