Implement REINFORCE Leave-One-Out (RLOO) advantages and the corresponding policy-gradient loss (Ahmadian et al., 2024) — a strong baseline for RL fine-tuning of LLMs that needs no learned critic.
Advantages. Given a group of sampled responses for the same prompt with rewards :
If , return [0.0]. Round each advantage to 6 decimals.
Loss (minimization form). With sequence log-probs :
Optionally shape rewards with a KL penalty before computing advantages:
where kl_log_ratios_i ≈ log πθ − log πref.
Implement:
rloo_advantages(rewards) -> list[float]rloo_loss(log_probs, rewards, kl_log_ratios=None, beta=0.0) -> float (6 decimals)