ORPO: Odds Ratio Preference Optimization

ORPO (Odds Ratio Preference Optimization) is a single-stage alignment method that combines supervised fine-tuning and preference optimization into one training objective. Instead of first training the model with SFT and then aligning it with DPO, ORPO does both simultaneously by adding an odds ratio penalty to the standard language modeling loss.

The odds ratio component penalizes the model for assigning higher probability to rejected responses relative to chosen ones. This is computed using the ratio of output probabilities, weighted by a lambda hyperparameter that controls the strength of the preference signal.

The traditional alignment pipeline requires two separate training stages: SFT first to teach the model to follow instructions, then DPO or another alignment method to learn preferences. Each stage requires its own hyperparameter tuning, monitoring, and compute budget.

ORPO collapses this into one stage, cutting total training time roughly in half. This also avoids the alignment tax where the DPO stage can degrade capabilities learned during SFT. By learning both objectives jointly, the model balances instruction following and preference alignment throughout training.

ORPO requires datasets that include both the instruction-following signal and preference pairs. Each example should contain a prompt, a chosen response (the preferred output), and a rejected response. The chosen response also serves as the SFT target, so it should be a high-quality instruction-following response.

This means your preference dataset needs to be more carefully curated than a standard DPO dataset, since the chosen responses must also work as SFT training examples. BIOS validates that your dataset contains all required fields and provides a preview showing the prompt, chosen, and rejected columns.

ORPO is most effective when you are starting from a base model (not yet SFT-trained) and want to produce an aligned model in a single run. It is also well-suited when compute budget is limited and you cannot afford the two-stage SFT + DPO pipeline.

ORPO works best with moderate-sized datasets (10K-50K examples) where the preference pairs are clearly differentiated. For very large datasets or when you need precise control over the SFT and alignment stages separately, the traditional two-stage approach may still be preferable.

Select ORPO as the training method in the BIOS configurator. The key hyperparameter is lambda, which controls the weight of the preference objective relative to the SFT loss. Default is 0.1, which works well for most tasks. Higher values emphasize preference alignment more strongly.

The BIOS training wizard configures ORPO-specific parameters including the lambda weight:

Monitor both the SFT loss and the odds ratio loss in the BIOS dashboard. A healthy training run shows both losses decreasing, with the SFT loss typically converging faster than the preference loss.

Because ORPO combines SFT and alignment in a single stage, your dataset has to do two jobs at once. The chosen responses serve as both the SFT training targets and the positive preference examples, so they need to be high-quality instruction-following responses that also represent the preferred style and quality level. Rejected responses should be clearly inferior but still coherent. Extremely low-quality rejections provide a weak learning signal.

The lambda hyperparameter is the most important tuning lever for ORPO. A lambda of 0.1 provides a moderate preference signal that works well for most tasks. If you find that the model follows instructions well but does not exhibit strong enough preference alignment, increase lambda to 0.2 or 0.3. If the model struggles to follow basic instructions, the preference signal may be too strong. Reduce lambda to 0.05.

ORPO typically benefits from slightly more training data than a separate SFT stage would require, because the model needs to learn both instruction following and preference alignment from the same examples. Aim for at least 10,000 high-quality examples. Below this threshold, the model may not see enough preference contrast to learn meaningful alignment, even if the SFT signal is sufficient.

Monitor both loss components in the BIOS dashboard. The SFT loss should decrease steadily throughout training. The odds ratio loss may be more volatile initially but should show a clear downward trend. If the odds ratio loss plateaus while the SFT loss continues to decrease, the model is learning to follow instructions but not picking up the preference signal. Consider increasing lambda or reviewing the quality of your preference pairs.

ORPO is particularly attractive for teams that want aligned models with minimal pipeline complexity. Instead of managing separate SFT and DPO training stages, you run a single ORPO job that produces an aligned model directly from a base model. This reduces total training time, simplifies experiment tracking, and avoids the alignment tax where preference optimization can degrade capabilities learned during SFT.

To get started, prepare a dataset where each example contains a prompt, a high-quality chosen response, and a clearly inferior rejected response. The chosen responses should be the quality you want the model to produce. They serve as both the instruction-following target and the positive preference example. Upload the dataset to BIOS and use the preview feature to verify the pairs look correct.

Create a new training run with ORPO as the method. Select a base model (not an SFT-trained model, since ORPO handles the SFT stage internally). Set lambda to 0.1, learning rate to 5e-6, and train for 2 to 3 epochs. BIOS will display both the SFT loss and the preference loss in real time. After training completes, evaluate the model to confirm it both follows instructions correctly and produces outputs aligned with the preferred style.