Adapter Types Compared: LoRA, QLoRA, Full Fine-Tune and Beyond

Adapters are small trainable modules inserted into a frozen pretrained model. They allow fine-tuning with a fraction of the memory and compute required for full parameter training. The family of adapter methods has grown quickly, and each one trades off efficiency, expressiveness, and compatibility differently.

USF BIOS supports multiple adapter types so you can match the method to your constraints. The choice depends on available VRAM, dataset size, required quality, and inference deployment requirements.

LoRA injects low-rank matrices (A times B) into transformer layers, typically targeting attention projections (q, k, v, o) and MLP layers. With rank 16, a 7B model adds about 10-20M trainable parameters. Memory: approximately 16GB for 7B. Quality: excellent for most tasks. Speed: fast training, zero inference overhead after merging.

LoRA is the default recommendation for most fine-tuning scenarios. It balances quality, speed, and memory efficiency well, and produces adapters that can be merged into the base weights for deployment without any architectural changes.

QLoRA quantizes the frozen base model to 4-bit NormalFloat precision and trains LoRA adapters in full precision on top. Memory: approximately 6-8GB for 7B. Quality: nearly identical to LoRA. Speed: 10-20% slower due to dequantization during forward pass.

Choose QLoRA when VRAM is the primary constraint. It enables fine-tuning 70B models on a single 48GB GPU, which would require 4+ GPUs with full precision LoRA.

Full fine-tuning updates all parameters. Memory: approximately 56GB for 7B (4x model size for weights plus optimizer plus gradients). Quality: highest ceiling with sufficient data. Speed: comparable to LoRA per step, but often requires fewer total steps.

Reserve full fine-tuning for cases where adapter methods underperform: domain shifts requiring deep knowledge changes, very large datasets (100K+ examples), or when you need to modify the model fundamental capabilities.

AdaLoRA dynamically allocates the rank budget across layers based on importance scores during training. Layers that benefit from higher rank get more capacity, while less important layers use lower rank. This can achieve better quality than fixed-rank LoRA with the same total parameter budget.

AdaLoRA adds slight overhead for the importance scoring mechanism but can improve results on tasks where different layers contribute unevenly to the output quality.

LoHa (Low-Rank Hadamard) uses the Hadamard product of two low-rank matrices instead of their matrix product, providing a different parameterization that can be more expressive per parameter. BOFT (Butterfly Orthogonal Fine-Tuning) constrains weight updates to be orthogonal transformations using butterfly matrices, preserving pretrained weight norms and preventing catastrophic forgetting. ReFT (Representation Fine-Tuning) operates on hidden representations rather than weights, intervening in the forward pass to steer activations.

These are newer methods with narrower use cases. LoHa originated in image generation and can help when LoRA at a given rank is insufficient. BOFT is promising for continual learning. ReFT is particularly parameter-efficient but requires careful layer selection.

Start with LoRA (rank 16) as the default. If VRAM is tight, switch to QLoRA. If quality is insufficient with LoRA rank 64, consider full fine-tuning if you have sufficient data and compute. Try AdaLoRA if you want automatic rank allocation without manual experimentation.

The BIOS training wizard lets you select your adapter type and configure method-specific parameters:

On USF BIOS, comparing adapters is quick: run short training jobs with different adapter types on the same dataset and compare the loss curves and evaluation metrics.

View all your adapter comparison experiments in the BIOS training list:

Comparing adapter types requires a systematic approach to isolate the effect of the adapter from other variables. The most reliable method is to run controlled experiments where only the adapter type changes while all other parameters (dataset, base model, learning rate schedule, batch size, and number of epochs) remain constant. This is straightforward on BIOS because you can create multiple training runs with identical configurations except for the adapter selection.

Start by running three baseline experiments: LoRA with rank 16, QLoRA with rank 16, and LoRA with rank 64. These three configurations cover the most common tradeoff points: standard efficiency, maximum memory savings, and high-capacity adaptation. Compare the final validation loss and any task-specific evaluation metrics across all three runs. If LoRA rank 16 meets your quality requirements, there is no need to explore further. If it falls short, try LoRA rank 64 before moving to full fine-tuning.

When comparing results, look beyond just the final loss value. Check the shape of the loss curves. A smooth, steadily decreasing curve indicates stable training, while oscillations or plateaus may indicate suboptimal hyperparameters. Also compare inference quality on a held-out test set using task-specific metrics. Two runs with similar final loss values can produce noticeably different output quality on real-world examples.

Per-second billing keeps adapter comparison cheap. Each short comparison run (100 to 500 steps) costs only minutes of compute time. Running five adapter configurations as quick comparison experiments is far cheaper than committing to one configuration and discovering it was suboptimal after a full training run.

BIOS supports all the major adapter types in one place, so trying a different method on the same dataset is a small configuration change rather than a new pipeline. To get started, upload your dataset and navigate to the Training section. Click New Training Run and select your base model from the 250+ supported models.

In the adapter selection step, choose your method. For LoRA and QLoRA, you will configure rank, alpha, target modules, and dropout. For full fine-tuning, these adapter-specific parameters are replaced with the core training hyperparameters. For AdaLoRA, you set an initial rank budget that the algorithm distributes across layers during training. The BIOS configurator provides default values and inline documentation for all parameters.

A practical workflow for adapter selection is to start with a quick LoRA rank 16 run as your baseline. If the results are satisfactory, you are done. LoRA rank 16 is the sweet spot for most tasks. If quality is insufficient, increase rank to 32 or 64. If memory is the constraint, switch to QLoRA. Only if LoRA at high rank still falls short should you move to full fine-tuning, and only if you have a sufficiently large dataset (10,000+ examples) to justify the additional compute cost.

All your training runs are tracked in the BIOS training list with their adapter type, hyperparameters, and metrics clearly displayed. A glance at the list is enough to compare results across adapter types and identify the best configuration for your specific task and constraints.