Techniques

Schedule-free optimization

Technique · Training · 1 item since May 2026

Optimizers that drop the learning-rate schedule, often by averaging weights over training.

Trends

Items

  1. May 18

    ScheduleFree+ drops learning-rate schedules for pretraining

    A schedule-free optimizer that replaces the learning-rate schedule with momentum-based model averaging cuts training time to equivalent loss by 31% versus Warmup-Stable-Decay at 1,000 tokens per parameter and allows anytime stopping.