The H-Net paper trains with warmup-stable-decay: 10% linear warmup, a long constant phase, then 20% inverse-square-root decay. It cites Ibrahim et al. (2024), "Simple and Scalable Strategies to Continually Pre-train Large Language Models", which defines the decay on an interval from to (their name for the end point is ):
The H-Net paper states neither nor the final learning rate, so both are arguments here.
wsd_lr(step, total_steps, peak_lr, warmup_frac=0.1, decay_frac=0.2, end_lr=0.0, alpha=100.0)
w be warmup_frac times total_steps, rounded with Python's round, and d the same for decay_frac. The decay starts at = total_steps - d and ends at = total_steps.step < w: return peak_lr * step / w.d == 0 or step < total_steps - d: return peak_lr.step >= total_steps returns end_lr.make_wsd_scheduler(optimizer, total_steps, warmup_frac=0.1, decay_frac=0.2, end_frac=0.0, alpha=100.0)
torch.optim.lr_scheduler.LambdaLR that multiplies each parameter group's own learning rate by this schedule. A group's rate therefore peaks at its initial value and ends at end_frac times it.LambdaLR multiplies by whatever the function returns, so you can reuse wsd_lr with a peak of 1.