Low-communication distributed training

Technique · Training · 1 item since Apr 2026

Training across distant clusters that sync rarely, as in DiLoCo, instead of every step.

Organizations: Google DeepMind

Trends

Frontier labs secure compute beyond the traditional hyperscalers

Items

Apr 23
Google DeepMind's Decoupled DiLoCo
Fully asynchronous training across independent learner islands cuts inter-datacenter bandwidth from 198 Gbps to 0.84 Gbps, keeps 88% goodput under heavy hardware failures versus 58% for elastic data parallelism, and pre-trained a 12B model across four US regions over standard WAN.