Experiences with local models for coding
TLDR Dev
A developer tested small language models running locally on their machine for agentic coding tasks, evaluating models like Qwen 3.6 35B and Gemma 4 across a viability funnel covering RAM usage, speed, tool calling, and code quality. In automated evaluation of a task involving access log visualization, Qwen 35B MoE failed 5 out of 7 times on a 48GB machine but only once on a 64GB machine, suggesting memory significantly impacts output quality despite identical model settings. The experience revealed that task characteristics like required code search, number of files to edit, instruction specificity, and tech stack heavily influence whether local models remain viable alternatives to larger cloud-based models.
Why it matters
Recent experiences with locally run models for agentic coding reveal both challenges and successes, with these models handling straightforward coding tasks effectively but showing variable performance overall.
