Attention Is All You Need trains with Adam and a learning rate that rises linearly for the first warmup_steps updates and then decays with the inverse square root of the step number:
Implement two functions:
transformer_lr(step, d_model, warmup_steps, factor=1.0)
step is an integer (the first update is step 1, never 0).lr_schedule(num_steps, d_model, warmup_steps, factor=1.0)
1, 2, ..., num_steps in order, each value rounded to 8 decimals.The peak of the schedule is reached at step == warmup_steps, where both arguments of the min are equal; for d_model = 512 and warmup_steps = 4000 the peak is about .
x ** -0.5 is and x ** -1.5 is .step == warmup_steps; before it the warmup term is smaller, after it the decay term is smaller.range(1, num_steps + 1).