Harness optimization

Related

Organizations

MeituanMicrosoftNVIDIAScale AI

Techniques

AI controlAutomated AI researchCurriculum learningKnowledge distillationPrompt optimization

Trends

• Agents improve themselves through skills, harnesses, and memory
• Offensive AI cyber risk moves from evaluations into the wild

Items

1. Oct 9

   Building LLM agent systems like neural networks

   Treats retrieval, memory, and prompting modules as neural-network layers with prompt optimization as backpropagation and a NAS-equivalent search over configurations, reporting at least 5% gains from automatic prompt optimization and 11% from architecture search.

2. Oct 2

   Microsoft's ActiveSaddler

   Formulates agent-harness optimization as automated curriculum learning, modeling the evolving training curriculum as a non-stationary bandit over reusable failure-pattern arms, and lifts test Pass@1 by 4.4 points on GAIA2 and 7.5 points on Terminal-Bench 2.0 over a fixed scenario order.

3. Sep 25

   Pistis pairs IDRL with auto-harnessing

   A 9B/27B multimodal model family trained with Interleaved Distillation and Reinforcement Learning, paired with a system-level method that automatically improves the inference harness without updating model weights.

4. Sep 25

   Growing Harness evolves agent scaffolds from failures

   A failure-guided paradigm that grows a reusable, executable agent harness from a strategy-free scaffold, cutting deployed LLM calls 76-92% while keeping 4B-120B models effective across browsing and web-agent benchmarks.

5. Sep 18

   NVIDIA's SoL-Pi automates harness research

   Scaling automated auto-research loops at the agent-harness layer yields four transferable mechanisms that match the Pi harness on the 51-task EdgeBench while cutting recorded token traffic 44.7-49.0% and API cost by about a third.

6. Sep 11

   EvoSafeHarness evolves safety harnesses for frozen agents

   Synthesizes a deployable, model- and domain-specific safety harness by jointly searching a natural-language policy and executable code, cutting DecodingTrust-Agent attack success from 45.6% to 10.0% and reaching 82.8% utility at 0.0% attack success on AgentDojo.

7. Aug 8

   Meituan's AutoDesign rewrites its own harness

   A meta-harness optimizer has a code agent recursively rewrite its harness from rollout feedback, lifting average PosterBench scores from 54.99 to 67.39 and beating the closed Claude Design system by 7.45 points with a frozen base model.

8. Aug 1

   Scale AI's HarnessOpt-Bench

   A benchmark of how well frontier LLMs optimize an agent's harness (prompts, tools, control flow, memory) under a fixed budget, finding across 111 scored runs that the optimizer model matters more than the coding harness it acts through.