Inside the Together AI kernels team

Together AI · 3 months ago

Together AI's kernels team, led by Dan Fu and Tri Dao, develops GPU optimization software that bridges the gap between AI models and hardware efficiency. The team achieved a 3.6x speedup for a real-time voice agent company, reducing latency from 281ms to 77ms on Llama-3.2-1B, and created ThunderKittens, a library that reduced CUDA code from 1,000+ lines to 100-200 lines for adapting kernels to new NVIDIA Blackwell GPUs. This kernel optimization work directly impacts production AI systems by determining inference costs, training time, and whether AI applications feel responsive to end users.