Warmup, Learning Rate Warmup

August 23, 2025 2 weeks ago 1 min read

Schedule that gradually increases the learning rate at the start of training to stabilize optimization, commonly combined with cosine decay or step schedules.