SimPO: Simple Preference Optimization Without Reference Models

SimPO (Simple Preference Optimization) is an alignment algorithm that trains language models to prefer human-chosen responses over rejected alternatives, similar to DPO, but without requiring a reference model during training. This eliminates the need to keep a frozen copy of the base model in memory, roughly halving the VRAM requirement compared to DPO.

SimPO achieves this by using the average log probability of the generated sequence as an implicit reward, rather than computing the KL divergence from a reference policy. The length-normalized reward prevents the model from gaming the objective by producing shorter responses.

DPO requires keeping both the training model and a frozen reference model in memory. For a 7B parameter model, DPO needs approximately 28GB just for the two model copies before accounting for optimizer states. SimPO needs only one model copy, cutting this to 14GB.

Despite the simplified objective, SimPO matches or exceeds DPO on standard alignment benchmarks including MT-Bench and AlpacaEval. The length normalization in SimPO also produces more consistent response lengths, avoiding the verbosity that DPO models sometimes exhibit.

SimPO uses the same paired preference dataset format as DPO: each example contains a prompt, a chosen response, and a rejected response. The quality of these pairs is critical. The chosen response should be clearly better than the rejected one, not just marginally different.

A typical alignment dataset contains 5,000 to 50,000 preference pairs. Smaller datasets risk overfitting, while very large datasets may not improve quality proportionally. BIOS validates your preference dataset format during upload and shows a preview of the pairs before training begins.

Choose SimPO over DPO when VRAM is constrained and you want to align larger models on smaller GPUs. SimPO is particularly effective when your preference data is clean and the quality gap between chosen and rejected responses is clear.

SimPO is also a good default when you are unsure which alignment method to use. Its simplicity means fewer hyperparameters to tune, and the reference-free training is faster per step. On BIOS, SimPO training typically completes 15-25% faster than equivalent DPO runs.

Select SimPO as the training method in the BIOS configurator. Key hyperparameters include beta (controls preference strength, default 2.0), gamma (length bonus coefficient, default 0.5), and the standard training parameters like learning rate and batch size.

The BIOS training wizard shows SimPO-specific configuration options when you select this method:

BIOS handles the reward computation internally, applying the length-normalized log probability objective. You can monitor the reward margins, accuracy on the preference pairs, and the overall training loss through the live metrics dashboard.

The training detail view streams SimPO metrics in real time so you can track alignment progress:

SimPO benefits from many of the same practices as DPO, with a few method-specific considerations. Since SimPO does not use a reference model, the beta parameter plays a different role than in DPO. It directly controls the strength of the preference optimization rather than regulating divergence from a reference distribution. Start with the default beta of 2.0 and adjust based on your results. If alignment is too weak, increase beta. If the model becomes overly constrained or repetitive, decrease it.

The gamma parameter controls the length bonus and is unique to SimPO. A positive gamma value rewards longer responses, while a lower gamma encourages conciseness. For tasks where response length matters, such as detailed explanations or thorough answers, a gamma of 0.5 to 1.0 works well. For tasks where conciseness is preferred, reduce gamma to 0.1 or 0.0.

Because SimPO uses length-normalized rewards, your preference dataset should ideally contain chosen and rejected responses of similar lengths. If chosen responses are systematically much longer or shorter than rejected ones, the length normalization can interfere with the preference signal. Review your dataset pairs to ensure the preference quality difference is not confounded by length differences.

SimPO trains faster than DPO per step because it does not need to compute log probabilities under a reference model. Take advantage of this speed by running more experiments with different hyperparameter configurations.

SimPO takes the same inputs as DPO: a preference dataset in JSONL with prompt, chosen, and rejected fields. Upload it through the Datasets page and verify the format with the preview feature. Create a new training run, select SimPO as the method, and point it to your SFT-trained base model.

Start with the default hyperparameters (beta 2.0, gamma 0.5, learning rate 5e-7) and train for 1 to 2 epochs. Monitor the reward margins and preference accuracy in the BIOS dashboard. SimPO typically converges faster than DPO, so you may see good results after just 1 epoch. Evaluate the aligned model on your test set and compare against your SFT baseline to measure the improvement.

If you previously ran DPO and want to compare, BIOS keeps all your training runs in the training list so you can view metrics from both methods side by side. A comparison experiment costs very little, since you pay only for the seconds of compute it uses.