Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize

MarkTechPost · 1 month ago · 33

HarnessDev tested whether LLMs can write and iteratively improve the agent “harness” (the runnable code with tools and execution loop) rather than just produce answers. Only 34 out of 64 harness changes moved held-out scores in the same direction as feedback, with execution gains described as small and noisy. The work concludes harness quality is executor-specific and that much generated state/memory code often never runs, so generalization across executors and revisions is limited.