VLM Fine-Tuning: How to Train Vision-Language Models

Vision-language models (VLMs) are multimodal neural networks that process both images and text. They combine a vision encoder (typically based on CLIP, SigLIP, or a similar contrastive vision model) with a large language model through a projection layer that maps visual features into the language model embedding space.

This architecture allows VLMs to answer questions about images, describe visual content, extract text from documents via OCR, analyze charts and diagrams, and reason about visual scenes. Fine-tuning VLMs lets you specialize these capabilities for your specific visual domain and output requirements.

USF BIOS supports the major VLM families: InternVL 2 and InternVL 2.5 (state-of-the-art open VLMs with strong OCR and reasoning capabilities), Qwen-VL (Alibaba multimodal model family), LLaVA (visual instruction tuning architecture from Microsoft Research), DeepSeek-VL (efficient multimodal model), and MiniCPM-V (compact VLM for resource-constrained deployments).

Each architecture has different strengths. InternVL excels at document understanding and complex visual reasoning. Qwen-VL offers strong multilingual visual capabilities. LLaVA is well-suited for visual instruction following. The choice depends on your use case, model size requirements, and deployment constraints.

VLM datasets pair images with text annotations. For SFT, each example contains an image (path or URL), a text instruction or question about the image, and the expected text response. For preference alignment, add chosen and rejected responses for the same image-question pair.

Images can be provided as file paths (if uploaded alongside the dataset), URLs (for remote images), or base64-encoded strings in the JSONL. BIOS handles image loading, resizing to the model native resolution, and encoding automatically. The platform validates image accessibility and format during dataset upload.

BIOS supports the same training methods for VLMs as for text-only LLMs: SFT, LoRA, QLoRA, and full fine-tuning. The adapter is applied to the language model component, while the vision encoder can be frozen or fine-tuned depending on your configuration.

Freezing the vision encoder is recommended for most tasks because it preserves the visual feature extraction capabilities learned during pretraining. Only unfreeze the vision encoder if your images are fundamentally different from what the model saw during pretraining (e.g., medical imaging, satellite imagery, or specialized scientific visualizations).

Teams fine-tune VLMs for a wide range of visual tasks:

• Document understanding: extract structured data from invoices, receipts, forms, and contracts.
• Visual question answering: answer questions about product images, real estate photos, or technical diagrams.
• OCR and text extraction: read text from images in your specific domain and format.
• Medical imaging: interpret X-rays, pathology slides, or dermatological images (with appropriate data handling).
• Quality inspection: identify defects in manufacturing images.
• Chart analysis: read and interpret business charts and graphs.

Select a VLM architecture in the BIOS model picker (filtered by the VLM tag). Upload your image-text dataset in the supported format. The configurator automatically adjusts for VLM-specific settings: image resolution, vision encoder handling, and multi-image support where applicable.

The BIOS training wizard adapts its interface for VLM-specific configuration:

VLM training requires more VRAM than text-only training due to the vision encoder parameters and image token sequences. BIOS shows accurate VRAM estimates for VLM jobs and recommends appropriate GPU tiers. QLoRA is particularly useful for VLM fine-tuning, as it allows training larger VLMs on single GPUs.

Monitor your VLM training progress with real-time loss curves and metrics:

VLM fine-tuning has several unique considerations compared to text-only LLM training. The most important is image quality and consistency. Ensure your training images are clear, properly oriented, and representative of the images your model will encounter in production. Blurry, low-resolution, or incorrectly rotated images introduce noise that degrades model quality. If your use case involves document processing, include documents with the same layouts, fonts, and quality levels that the production system will handle.

Freeze the vision encoder for most fine-tuning tasks. The vision encoder was pre-trained on a large, diverse image dataset and has learned reliable visual feature extraction. Unfreezing it during fine-tuning risks overwriting these general visual capabilities in favor of narrow patterns from your smaller dataset. Only unfreeze the vision encoder if your images are fundamentally different from natural images, such as medical scans, satellite imagery, or specialized scientific visualizations, where the pre-trained features may not transfer well.

Dataset size requirements for VLM fine-tuning are generally higher than for text-only LLMs because the model needs to learn both visual patterns and text-image associations. Aim for at least 5,000 image-text pairs for LoRA fine-tuning and 20,000 or more for full fine-tuning. If your dataset is smaller, use aggressive data augmentation (image flipping, rotation, color jittering) to artificially increase the effective dataset size.

Use QLoRA as the default adapter for VLM fine-tuning. The vision encoder adds significant memory overhead, and QLoRA quantization of the language model component frees up memory for the visual processing pipeline. On BIOS, QLoRA typically allows you to fine-tune VLMs that are one tier larger than what standard LoRA would support on the same hardware.

The first step toward a fine-tuned VLM is an image-text dataset. Each example should contain an image (as a file path, URL, or base64-encoded string), a text instruction or question about the image, and the expected text response. Format the dataset as JSONL and upload it through the Datasets page. BIOS validates the image references and text fields during upload.

In the Training section, create a new training run and filter the model list to show VLM architectures. Select your target model: InternVL for document understanding, Qwen-VL for multilingual visual tasks, LLaVA for visual instruction following, or MiniCPM-V for resource-constrained deployments. Choose QLoRA as the adapter type for the best balance of memory efficiency and quality.

Configure the training hyperparameters using the BIOS defaults as a starting point. Launch the job and monitor the loss curves in the dashboard. VLM training typically shows a steeper initial loss decrease than text-only training because the model is learning to associate visual features with text descriptions. After training completes, evaluate the model by generating responses to test images and comparing the quality against the base VLM.