Trends

Agent benchmarks expose reliability gaps and lose trust

Agent evaluation and reliability · Apr 25 – Jul 10

Organizations

OpenAIAnt GroupArtificial AnalysisIBMUniversity of Illinois Urbana-ChampaignYonsei University

Updates

  1. Jul 10

    OpenAI's datapoint-level audit found breaking issues in about 27-34% of SWE-Bench Pro tasks and retracted its recommendation to adopt the benchmark, yet xAI still headlined Grok 4.5 with a 64.7% SWE-Bench Pro score the same week.

    OpenAI finds ~30% of SWE-Bench Pro broken

    Against xAI's Grok 4.5

  2. Jun 5

    A benchmark of 474 executable games found frontier models solid on clean tasks (Gemini-3.1-pro 74.2%) but falling to 57.6% under counterfactual revision and pruning evidence they needed.

    Executable games expose weak belief revision

  3. May 29

    CausaLab found GPT-5.2-high reaching 92% task accuracy but only 0.471 F1 on the causal graph, ITBench-AA put every frontier model below 50% on enterprise IT operations, and LaRA showed RL post-training defeats output-level contamination detectors.

    UIUC's CausaLab tests causal discoveryFrontier models score below 50% on ITBench-AAYonsei's LaRA detects RL training contamination

  4. Apr 25New

    OpenAI pitched GPT-5.5 at coding tasks with a median human effort of 20 hours and led with agentic results: a record 82.7% on Terminal-Bench 2.0 and 84.9% on GDPval.

    OpenAI's GPT-5.5

Predictions

  1. Oct 2026

    Long-horizon agentic metrics such as SWE-Bench Pro, Terminal-Bench, GDPval, and Expert-SWE replace static reasoning evals as the main frontier comparison, pulling training objectives toward multi-session autonomous engineering. Apr 25

  2. Nov 2026

    At least two frontier labs publish structured evaluation frameworks for multi-agent reliability beyond single-turn benchmarks, driven by liability from agentic IT, legal, and financial workflow failures. May 29

  3. Dec 2026

    Agent training pipelines adopt grounded, dependency-aware data synthesis and belief-revision-specific evaluation as static single-turn benchmarks fail to predict reliability in interactive deployment. Jun 5

  4. Jan 2027

    Agent-assisted dataset auditing and independently built, contamination-resistant evaluations become procurement and safety-case requirements, and vendor-reported scores face routine demands for third-party replication. Jul 10