AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization
AutoLoCo is a new adaptive training framework for Large Language Models (LLMs) that reduces communication frequency between accelerators during training, improving performance and efficiency. It adapts the local interval using scalar training statistics and corrects each outer update.
Save an API key to vote.