KTO: Kahneman-Tversky Optimization for AI Alignment

KTO (Kahneman-Tversky Optimization) is an alignment method grounded in prospect theory from behavioral economics. Unlike DPO and SimPO, which require paired preferences (chosen vs rejected for the same prompt), KTO works with single-response binary feedback: each example is labeled as either good (thumbs up) or bad (thumbs down), independently.

This is significant because paired preference data is expensive to collect. Most real-world feedback systems produce binary signals. Users click thumbs up or thumbs down on individual responses, they do not compare two alternatives side by side. KTO lets you use this naturally occurring feedback format directly.

Prospect theory, developed by Daniel Kahneman and Amos Tversky, describes how people evaluate gains and losses asymmetrically. Losses are felt more strongly than equivalent gains. Losing $100 feels worse than gaining $100 feels good. KTO applies this principle to model training.

The KTO loss function weights negative examples (bad responses) more heavily than positive examples (good responses), matching the psychological reality that users care more about avoiding bad outputs than about maximizing good ones. This loss asymmetry makes KTO particularly effective at reducing harmful or low-quality outputs.

KTO datasets contain three fields: a prompt, a response, and a binary label (desirable or undesirable). There is no pairing. Each example stands alone. This makes data collection straightforward: log model outputs, have annotators (or automated systems) label each one as good or bad, and train.

The ratio of desirable to undesirable examples affects training. KTO works well with ratios between 1:1 and 3:1 (desirable to undesirable). The algorithm internally adjusts for class imbalance, but extremely skewed ratios (10:1 or worse) may require tuning the beta hyperparameter.

KTO is the natural choice when your feedback data is binary rather than paired. If you have user thumbs-up/down logs, helpfulness ratings converted to binary, or automated quality scores thresholded into good/bad, KTO lets you use this data directly without the artificial step of creating preference pairs.

KTO also works well when you want to improve safety. Since the loss function weights bad examples more heavily, KTO is effective at teaching models to avoid specific failure modes. Collect examples of undesirable behavior, label them as bad, pair them with good examples, and train. The model learns to steer away from the negative patterns.

Select KTO as the training method in the BIOS configurator. Upload your dataset with prompt, response, and label columns. BIOS validates the format and shows the distribution of desirable vs undesirable examples.

The BIOS training wizard configures KTO-specific parameters including the beta weight and loss asymmetry:

The main hyperparameter is beta, which controls the strength of the KTO loss (default 0.1). Higher beta values produce stronger alignment but may degrade general capabilities. Monitor the desirable and undesirable loss components separately in the BIOS dashboard to ensure both are decreasing.

KTO training benefits from several specific practices that differ from paired preference methods. The most important is maintaining a balanced ratio of desirable to undesirable examples. KTO works best with ratios between 1:1 and 3:1 (desirable to undesirable). Extremely skewed ratios, such as 10:1 desirable to undesirable, can cause the model to underweight the negative signal, resulting in weak alignment against undesirable behaviors.

The loss asymmetry in KTO means that the model learns more strongly from undesirable examples than from desirable ones, reflecting the prospect theory insight that losses loom larger than gains. This makes KTO particularly effective at teaching the model what not to do. When building your dataset, pay special attention to the undesirable examples. They should represent the specific failure modes you want the model to avoid, such as hallucinations, harmful content, or off-topic responses.

For the beta hyperparameter, start with the default of 0.1 and adjust based on evaluation. If the model still produces undesirable outputs after training, increase beta to strengthen the alignment signal. If the model becomes overly cautious or refuses appropriate requests, beta may be too high. Reduce it to give the model more freedom. KTO is generally less sensitive to hyperparameter choices than DPO because the loss function has fewer moving parts.

KTO is uniquely valuable for safety alignment because the loss asymmetry naturally emphasizes avoiding bad outputs. If your primary goal is reducing harmful, incorrect, or policy-violating responses, KTO with a dataset focused on undesirable behaviors can be more effective than DPO for this specific objective.

KTO is the easiest alignment method to get started with because it uses the simplest dataset format. You do not need to create paired preference comparisons. To begin, collect model outputs and label each one as desirable or undesirable. This labeling can come from human annotators, automated quality metrics, or production feedback signals like user thumbs-up and thumbs-down buttons.

Format your dataset as JSONL with three fields: prompt, response, and label (where label is either desirable or undesirable). Upload it to BIOS and verify the format and distribution with the preview feature. BIOS shows the ratio of desirable to undesirable examples, which helps you confirm the dataset is reasonably balanced.

Create a new training run, select KTO as the method, and configure the hyperparameters. Start with beta 0.1 and learning rate 5e-7. Train for 1 to 2 epochs and monitor the separate loss components for desirable and undesirable examples. Both should decrease during healthy training, with the undesirable loss typically showing a steeper decline due to the loss asymmetry.

After training, evaluate the model by generating outputs for your test prompts and checking whether the specific failure modes represented in your undesirable examples have been reduced. KTO is most effective when evaluated against the specific behaviors it was trained to avoid rather than on general-purpose benchmarks.