Trends

Alternatives to Transformers start winning at equal compute

Architecture and inference efficiency · May 15 – May 22

Organizations

University of Southern California

Updates

  1. May 22

    HRM-Text's 1B hierarchical recurrent model, trained on 40B tokens for $1,500, reached 60.7% MMLU, competitive with 2–7B Transformers trained on 100–900x more tokens, and ScheduleFree+ cut time to equivalent loss by 31% versus Warmup-Stable-Decay.

    HRM-Text pretrains a hierarchical recurrent modelScheduleFree+ drops learning-rate schedules for pretraining

  2. May 15New

    USC's Attractor Models let a 770M model match a 1.3B Transformer trained on 2x tokens and solve 91.4% of Sudoku-Extreme where frontier LLMs score 0%, while AIRA-Compose agents found 14 architectures, including AIRAformer-C, which scales 54% faster than Llama 3.2.

    USC's Attractor ModelsAgents discover new neural architectures

Predictions

  1. Nov 2026

    At least one frontier lab incorporates an implicit refinement mechanism (fixed-point, attractor, or convergence-based) into a production model architecture. May 15

  2. Nov 2026

    At least one frontier lab incorporates a non-standard architectural element, recurrent, hybrid, or AI-discovered, into a production model. May 22