Study: LLMs can build their own coding harnesses, imperfectly

Researchers from ByteDance Seed, SUTD, Georgia Tech, M-A-P, and TokenWave.AI built HarnessDev, a benchmark that scores the runnable software harness a model constructs rather than just its final answer. Starting from a blank harness, six creator large language models (LLMs) built and evolved harnesses across 5 benchmarks and 2,207 tasks; the self-built harnesses matched human references on writing and machine-learning experimentation tasks but lagged on coding and search, and only 34 of 64 evolution changes generalized to held-out tasks.

Why it matters: This is a concrete data point in the broader push toward self-improving AI agents: it shows current models can bootstrap useful tooling for themselves in some domains, but that their self-modifications don't reliably transfer — a limiting factor for claims of open-ended agent self-improvement.