CPO: Contrastive Preference Optimization for LLM Alignment

CPO (Contrastive Preference Optimization) uses a contrastive learning objective to align language models with human preferences. Unlike DPO which focuses on the relative probability between chosen and rejected responses, CPO explicitly maximizes the absolute probability of chosen responses while minimizing the probability of rejected ones.

This contrastive framing prevents a common failure mode in DPO where the model can satisfy the relative objective by simply reducing both probabilities, as long as the chosen response probability remains higher. CPO maintains high generation quality for the preferred response while steering away from undesirable outputs.

The CPO loss combines two terms: a behavior cloning loss on the chosen response (encouraging the model to reproduce preferred outputs) and a contrastive margin loss between chosen and rejected responses (encoding the preference signal). The behavior cloning term acts as a regularizer that prevents the model from drifting too far from high-quality generation.

The balance between these two objectives is controlled by a hyperparameter. A higher weight on the contrastive term produces stronger alignment, while a higher weight on the behavior cloning term preserves general capabilities. CPO can operate without a reference model, similar to SimPO, reducing memory requirements compared to DPO.

CPO uses the standard paired preference format: prompt, chosen response, and rejected response. The quality requirements are similar to DPO. Clear preference signals between chosen and rejected responses lead to better training outcomes.

CPO tends to be more tolerant of noisy preference labels than DPO because the behavior cloning component on chosen responses provides a stable learning signal even when some preference pairs are ambiguous or mislabeled.

CPO is a strong choice when you want alignment without sacrificing generation quality. The behavior cloning term ensures the model continues to produce high-quality outputs for the preferred style, not just outputs that score higher than the rejected alternative.

CPO works particularly well for tasks where maintaining output quality is as important as preference alignment: creative writing, code generation, and technical documentation. These tasks require both following preferences (tone, format, accuracy) and maintaining high generative capability.

Select CPO as the training method in the BIOS configurator. The main hyperparameter is beta, controlling the contrastive loss strength. BIOS provides defaults tuned for common use cases. Upload your preference dataset and the platform validates the chosen/rejected format automatically.

The BIOS training configuration wizard shows CPO-specific parameters:

Monitor the chosen loss and contrastive loss separately in the dashboard. A healthy CPO run shows the chosen loss decreasing (model learning to reproduce preferred outputs) while the margin between chosen and rejected probabilities increases.

CPO benefits from high-quality chosen responses because the behavior cloning component trains the model to reproduce them directly. Unlike DPO, where the model only learns the relative preference between chosen and rejected, CPO explicitly trains the model to generate chosen-quality outputs. This means your chosen responses should be exemplary. They set the ceiling for what the model can produce after alignment.

The balance between the behavior cloning and contrastive objectives is controlled by the beta parameter. Start with the BIOS default and adjust based on evaluation results. If the aligned model produces high-quality outputs that are not sufficiently aligned with preferences, increase beta to strengthen the contrastive signal. If the model seems to lose generation quality after alignment, decrease beta to emphasize the behavior cloning component.

CPO is particularly well-suited for domains where maintaining output quality is critical: code generation, technical writing, and customer-facing content. In these domains, it is not enough for the model to prefer the right style. It must also maintain high generative capability. The behavior cloning regularizer in CPO provides this guarantee more explicitly than DPO, making CPO a strong default for quality-sensitive applications.

When building your preference dataset for CPO, invest extra effort in the quality of chosen responses. Since these responses also serve as behavior cloning targets, they should represent the best possible output for each prompt. Consider having senior annotators or domain experts create the chosen responses rather than using model-generated examples.

CPO training starts with the dataset: prompt, chosen, and rejected fields, where the chosen responses are high-quality examples of the output style you want the model to produce. Upload the dataset and verify the pairs using the BIOS preview feature. Create a new training run, select CPO as the method, and configure the beta parameter.

Start with a learning rate of 5e-7 and train for 1 to 2 epochs. Monitor the chosen loss and contrastive loss separately in the BIOS dashboard. Both should decrease during healthy training. The chosen loss reflects how well the model reproduces preferred outputs, while the contrastive loss reflects how well it distinguishes preferred from dispreferred responses.

After training, compare the CPO-aligned model against both the SFT baseline and any other alignment methods you have tried (DPO, SimPO). The BIOS training list keeps all your experiments accessible for comparison. CPO often produces the best results for quality-sensitive tasks where maintaining high generation capability alongside preference alignment is the primary goal.