RLHF Methods: Offline Alignment vs Online Reinforcement Learning

Reinforcement Learning from Human Feedback (RLHF) is the broad category of techniques for aligning language model behavior with human preferences. The original RLHF pipeline involved three steps: supervised fine-tuning (SFT), reward model training, and reinforcement learning (typically PPO). Since then, the field has split into offline methods that skip the RL step and online methods that preserve it.

Both approaches share the same goal: make the model produce outputs that humans prefer. They differ in how they encode and optimize for the preference signal, with significant implications for training complexity, compute cost, and alignment quality.

Offline methods optimize directly on a fixed dataset of preference pairs without generating new responses during training. DPO (Direct Preference Optimization) reformulates the RL objective as a binary classification problem on chosen vs rejected pairs. SimPO removes the reference model requirement. ORPO combines SFT and alignment in one stage. CPO adds a behavior cloning regularizer. KTO works with non-paired binary feedback.

All six offline methods are available on USF BIOS today. Their main advantage is simplicity: no reward model needed (except for Reward Modeling itself), no generation during training, and straightforward hyperparameter tuning. Training is stable and predictable.

The BIOS training wizard lets you select any of the six offline alignment methods and configures method-specific parameters automatically:

Online RL methods generate new responses during training and score them with a reward model, then use the reward signal to update the policy. PPO (Proximal Policy Optimization) is the classic algorithm, using a clipped surrogate objective to ensure stable policy updates. GRPO (Group Relative Policy Optimization) computes advantages relative to a group of sampled responses rather than a value function baseline. GKD (Generalized Knowledge Distillation) combines reward optimization with knowledge distillation from a teacher model.

These methods are coming soon to USF BIOS. They require a trained reward model (which you can already build on BIOS) and are more compute-intensive than offline methods, but they can discover and reinforce high-quality behaviors that are not represented in the static preference dataset.

The two families differ on three axes:

• Compute cost: offline methods process a fixed dataset and finish in a predictable time. Online methods generate and score responses during training, multiplying compute by 3-5x.
• Stability: offline methods have well-understood training dynamics. Online RL can be unstable, requiring careful reward model calibration and PPO hyperparameter tuning.
• Alignment quality: online methods can explore the response space and find high-quality outputs that the preference dataset does not contain. Offline methods are limited to the quality distribution of the training data.

For most production use cases, offline methods provide sufficient alignment quality. Online RL is most valuable when you need to push alignment quality beyond what your preference data captures.

Start with DPO if you have paired preference data. It is the most well-studied offline method with reliable results. Try SimPO if DPO VRAM requirements are too high. Use ORPO if you want to skip the SFT stage. Use KTO if you only have binary feedback. Train a reward model if you plan to use online RL when it becomes available on BIOS.

For most teams, the offline methods available today on BIOS provide excellent alignment quality. Online RL methods will offer an additional tool for teams that need to push the frontier of model quality and are willing to invest the additional compute and tuning effort.

You can compare results from different alignment methods in the BIOS training detail view:

Regardless of which alignment method you choose, several best practices apply universally. First, always start from an SFT-trained model. Alignment methods refine an already-capable model. They are not designed to teach the model basic instruction following. If your base model cannot produce coherent responses in your domain, alignment training will not fix that. Run SFT first, validate the model capabilities, and then apply alignment.

Second, invest in preference data quality. Nothing shapes alignment results more than the quality of your preference annotations, whichever method you pick. Clear, consistent, and informative preference labels produce good alignment. Noisy, contradictory, or trivially obvious labels produce weak alignment. If you are collecting preference data through human annotation, establish clear guidelines, train your annotators, and measure inter-annotator agreement.

Third, evaluate alignment on the specific dimensions you care about, not just general benchmarks. If you are aligning for safety, evaluate on safety-relevant test cases. If you are aligning for tone and style, evaluate on style-sensitive prompts. General-purpose alignment benchmarks like MT-Bench measure overall quality but may not capture the specific improvements your alignment training is targeting.

Fourth, do not over-align. Training for too many epochs on preference data can cause the model to become overly cautious, repetitive, or narrow in its outputs. Monitor reward accuracy during training. Once it exceeds 90 to 95 percent, additional training is likely overfitting to the preference patterns rather than improving genuine alignment. Stop training at the checkpoint with the best validation metrics, not at the end of the scheduled epochs.

BIOS covers alignment training more fully than any comparable platform, with all six offline methods (DPO, SimPO, ORPO, CPO, KTO, and Reward Modeling) in one interface. To get started, you need an SFT-trained model checkpoint and a preference dataset. If you are new to alignment training, DPO is the recommended starting method because it is the most well-studied and has reliable, predictable training dynamics.

Prepare your preference dataset in the appropriate format: JSONL with prompt, chosen, and rejected fields for DPO, SimPO, ORPO, and CPO, or prompt, response, and label fields for KTO. Upload the dataset through the Datasets page and verify the format with the preview feature. Create a new training run, select your alignment method, and configure the method-specific hyperparameters using the BIOS defaults as a starting point.

Launch the job and monitor the alignment-specific metrics in the BIOS dashboard. Each method reports different metrics. DPO shows reward margins and accuracy, SimPO shows length-normalized rewards, ORPO shows both SFT and preference losses, and KTO shows separate desirable and undesirable loss components. After training, evaluate the aligned model on your test set and compare against the SFT baseline to measure the improvement.

If you are unsure which method to choose, run quick comparison experiments with DPO and SimPO on the same dataset. These two methods have different tradeoffs (DPO uses a reference model for stability, while SimPO skips it for efficiency) and one may work better than the other on your specific data. Since each run bills by the second, these comparisons cost little.