jamesob's guide to running SOTA LLMs locally

GitHub · 3 months ago · 24

James Obermayer published a detailed technical guide for building local GPU systems to run state-of-the-art language models, ranging from $2,000 setups with RTX 3090s running Qwen models to $40,000+ configurations with 4× RTX 6000 Pro cards (384GB VRAM) running models approaching Claude Opus quality. The guide includes specific hardware bill-of-materials, BIOS configuration steps, kernel parameters, and Docker-based serving configurations, with the $40k system achieving 27.5 GB/s unidirectional GPU peer-to-peer bandwidth through custom PCIe Gen4 switches. Users can now deploy high-performance local inference with detailed hardware recommendations and ready-to-run software configurations instead of relying on cloud APIs.