Reward Modeling for RLHF: Training Custom Reward Functions

A reward model is a neural network trained to predict human preferences. Given a prompt and a response, the reward model outputs a scalar score indicating how good the response is according to human judgment. This learned reward function can then be used to optimize a language model via reinforcement learning.

Reward modeling is the first step in the classical RLHF (Reinforcement Learning from Human Feedback) pipeline: collect preference data, train a reward model, then use the reward model to guide RL training of the language model. While offline methods like DPO bypass the reward model entirely, explicit reward models remain valuable for online RL and for evaluating model quality.

In the full RLHF pipeline, the reward model serves as a proxy for human judgment. After training, it provides the reward signal for reinforcement learning algorithms like PPO (Proximal Policy Optimization), GRPO (Group Relative Policy Optimization), or GKD (Generalized Knowledge Distillation).

The quality of the reward model directly determines the quality of the RL-trained model. A reward model that assigns high scores to verbose but unhelpful responses will produce a verbose policy. This is why reward model evaluation is critical: you must verify that the reward model rankings correlate with actual human preferences before using it for RL training.

Reward model datasets use the same paired preference format as DPO: prompt, chosen response, rejected response. The reward model learns to assign higher scores to chosen responses and lower scores to rejected ones. The Bradley-Terry model is typically used to convert pairwise preferences into a pointwise reward.

Data quality matters more than quantity for reward models. Inconsistent labels (where annotators disagree on which response is better) introduce noise that degrades reward model accuracy. Aim for inter-annotator agreement above 70% and consider filtering out examples where annotators strongly disagree.

Select Reward Modeling as the training method in the BIOS configurator. Choose a base model. Reward models are typically initialized from an SFT model of the same architecture as your target policy model. Upload your preference dataset and configure the training hyperparameters.

The BIOS training wizard guides you through reward model configuration:

BIOS trains the reward model by replacing the language model head with a scalar output head. The training objective maximizes the margin between chosen and rejected response scores. The platform reports accuracy (percentage of pairs ranked correctly), mean reward for chosen vs rejected, and the reward margin distribution.

Monitor training progress and reward accuracy in real time on the BIOS dashboard:

USF BIOS is adding online reinforcement learning methods including PPO (Proximal Policy Optimization), GRPO (Group Relative Policy Optimization), and GKD (Generalized Knowledge Distillation). These methods will use your trained reward model to optimize the language model policy in real time, generating responses, scoring them with the reward model, and updating the policy to maximize the reward.

Online RL methods can achieve stronger alignment than offline methods because they explore the response space during training, discovering and reinforcing behaviors that the static preference dataset might not cover. We will announce availability here on the blog.

Building a reliable reward model requires careful attention to data quality and evaluation methodology. The most critical requirement is that your preference annotations are consistent. Inter-annotator agreement should be above 70 percent. If annotators frequently disagree on which response is better, the reward model will learn a noisy signal that produces unreliable scores. Consider using multiple annotators per example and taking the majority vote to improve label quality.

Reward model size matters for accuracy. The reward model should be large enough to capture the nuances of your preference criteria but does not need to be the same size as the policy model it will score. A 7B reward model can effectively guide training of a 7B or even 13B policy model. However, using a reward model much smaller than the policy model risks reward hacking, where the policy finds responses that score highly on the simpler reward model without actually being good.

Evaluate your reward model thoroughly before using it for RL training. Split your preference data into training and test sets. Report accuracy on the test set: how often the reward model correctly ranks the chosen response above the rejected response. Also inspect the reward score distribution: a healthy reward model produces well-separated distributions for chosen and rejected responses, not overlapping distributions that indicate the model cannot reliably distinguish quality levels.

Monitor for reward model overconfidence. If the reward model assigns very high scores to chosen responses and very low scores to rejected responses on the training set but performs poorly on the test set, it has overfit. Reduce training epochs or increase regularization to address this.

Two ingredients go into your first reward model: a preference dataset and an SFT-trained base model. The preference dataset should contain at least 5,000 pairs for a reliable reward model, with consistent annotations and clear quality differences between chosen and rejected responses. Upload the dataset through the Datasets page and verify the format with the preview feature.

Create a new training run, select Reward Modeling as the method, and choose your SFT-trained model as the base. BIOS automatically replaces the language model head with a scalar output head for reward prediction. Configure the learning rate (typically 1e-5 to 5e-5), batch size, and number of epochs (2 to 5 epochs is typical for reward models).

Launch the job and monitor the accuracy metric. This shows the percentage of preference pairs that the reward model ranks correctly. A well-trained reward model should achieve 75 percent or higher accuracy on the training set. If accuracy plateaus below 70 percent, your preference data may be too noisy or the model may be too small for the complexity of your preference criteria.

Once your reward model is trained and evaluated, you can use it as a quality evaluator for your other fine-tuned models. Score a set of responses from different model checkpoints to identify which checkpoint produces the highest-quality outputs according to your learned preference criteria. When online RL methods become available on BIOS, your trained reward model will be ready to guide PPO, GRPO, or GKD training directly.