Lehigh University

Trends

• Efficiency methods cut the compute and memory models need

Items

1. Apr 6

   MegaTrain trains 100B+ models on one GPU

   Notre Dame and Lehigh keep all parameters and optimizer states in host memory and stream them through the GPU, training models up to 120B parameters on a single H200 with 1.5TB of host RAM, at 1.84x the throughput of DeepSpeed ZeRO-3 offload at 14B scale.