Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize
MarkTechPost Asif Razzaq
HarnessDev tested whether LLMs can write and iteratively improve the agent “harness” (the runnable code with tools and execution loop) rather than just produce answers. Only 34 out of 64 harness changes moved held-out scores in the same direction as feedback, with execution gains described as small and noisy. The work concludes harness quality is executor-specific and that much generated state/memory code often never runs, so generalization across executors and revisions is limited.
Why it matters
ByteDance Seed, SUTD, Georgia Tech, M-A-P, and TokenWave.AI introduce HarnessDev, a benchmark that scores the runnable harness a model builds rather than the answer it returns. Starting from a seed that scores 0, 6 creator LLMs construct harnesses across 5 benchmarks and 2,207 tasks, then evolve them from execution feedback. Self-built harnesses match human references on writing and ML experimentation but trail on code and search, and only 34 of 64 evolution changes move the same direction on held-out tasks. The post Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize appeared first on MarkTechPost.