Continuing to pretrain a language model on new data needs a learning-rate schedule that re-warms from zero, decays again, and a data mixer that replays a fraction of the old corpus. Implement the two schedules and the mixer.
rewarm_cosine_lr(step, total_steps, warmup_steps, lr_max, lr_min)
step < warmup_steps: linear from 0 to lr_max, i.e. lr_max * step / warmup_steps.lr_max at step = warmup_steps to lr_min at step = total_steps - 1: with , return . Clamp p to .infinite_schedule_lr(step, warmup_steps, constant_steps, cooldown_steps, lr_max, lr_const, lr_min)
step < warmup_steps: linear from 0 to lr_max.constant_steps steps: cosine from lr_max down to lr_const (same formula with over this phase, where is the last step of the phase).cooldown_steps steps: linear from lr_const down to lr_min, reaching lr_min at the last cooldown step.lr_min.replay_mix_counts(batch_size, replay_frac, n_batches, rng)
n_batches batches, draw the number of replayed (old-corpus) examples as rng.binomial(batch_size, replay_frac). Return (counts, old_fraction): the list of counts and the realised fraction of old examples over all batches as a Python float.tokens_for_replay(new_tokens, replay_frac)
replay_frac of every batch is old data, return the number of old tokens consumed alongside new_tokens new ones: new_tokens * replay_frac / (1 - replay_frac).rng.binomial exactly once per batch so results are reproducible.