In Reinforcement Learning from Human Feedback (RLHF), a reward model is trained to predict human preferences between pairs of responses. Given a dataset of preference pairs where humans have indicated which response they prefer (chosen) over another (rejected), the reward model learns to assign higher scalar scores to preferred responses.
Implement a function that computes the reward model training loss and accuracy given arrays of reward scores for chosen and rejected responses.
The function should:
Your implementation should be numerically stable for both very large and very small score differences.