Continued Pre-Training: Domain Adaptation for Large Language Models

Continued pre-training (CPT) extends a pretrained model by training it further on a large corpus of domain-specific text using the standard next-token prediction objective. Unlike SFT, which teaches the model to follow instructions using structured input-output pairs, CPT injects raw knowledge into the model by exposing it to large volumes of unstructured domain text.

Think of CPT as extending the model pretraining phase with your own data. The model learns domain vocabulary, relationships between domain concepts, and the style of domain-specific writing. After CPT, the model has a stronger foundation in your domain, which makes subsequent SFT and alignment stages more effective.

Use CPT when the domain knowledge your model needs was underrepresented in the original pretraining data. General-purpose LLMs are trained primarily on web text, which means they have shallow coverage of specialized domains like clinical medicine, patent law, quantitative finance, or proprietary internal documentation.

If the model already understands your domain but needs to follow specific instructions or produce particular output formats, SFT alone is sufficient. But if the model lacks fundamental domain knowledge (it confuses medical terminology, misapplies legal precedents, or produces factually incorrect domain content), CPT is the right starting point.

CPT shows up most often in four domains:

• Medical: train on clinical notes, medical literature, drug databases, and treatment guidelines to build models that understand medical terminology and clinical reasoning patterns.
• Legal: train on case law, statutes, contracts, and regulatory filings to produce models that can analyze legal documents and reason about legal concepts.
• Financial: train on earnings reports, SEC filings, market analysis, and financial news to build models that understand financial instruments, valuation methods, and market dynamics.
• Code: train on specialized codebases, internal libraries, and documentation to build models that understand your tech stack and coding conventions.

CPT datasets are raw text files, with no instruction-response formatting needed. The text should be clean, well-structured, and representative of the domain knowledge you want the model to absorb. Common sources include: research papers, technical documentation, internal wikis, textbooks, standards documents, and curated web crawls filtered for domain relevance.

Quality matters more than quantity. A well-curated 1GB dataset of high-quality domain text often produces better results than 10GB of noisy web scrapes. Deduplicate your data, remove boilerplate (headers, footers, navigation text), and ensure the text is properly decoded (no encoding artifacts or broken characters).

Select Continued Pre-Training as the training method in the BIOS configurator. Upload your domain text as a plain text or JSONL file (one document per line). Key hyperparameters include learning rate (typically 1e-5 to 5e-5 for CPT, lower than SFT), context length, and number of training epochs.

The BIOS training wizard configures CPT-specific parameters:

CPT jobs are typically longer than SFT jobs because they process large text corpora. Monitor the perplexity on a held-out validation set in the BIOS dashboard. Decreasing perplexity indicates the model is learning the domain language patterns. Stop training when perplexity plateaus to avoid overfitting.

The recommended pipeline is: base model, then CPT on domain text, then SFT on instruction-response pairs, then optional alignment (DPO/SimPO/etc.) on preference data. Each stage builds on the previous one.

After CPT, the model understands your domain but does not yet follow instructions. The SFT stage teaches it to produce structured outputs in response to user queries. The alignment stage then refines output quality to match human preferences. On BIOS, you can chain these as separate training jobs, using each job checkpoint as the starting model for the next stage.

CPT results depend heavily on how you prepare and run the job. First, curate your corpus carefully. CPT is sensitive to data quality because the model absorbs patterns from every document in the training set. Include only high-quality, well-written domain text. Remove boilerplate content like headers, footers, navigation menus, copyright notices, and duplicate passages. If your source material includes tables, ensure they are formatted as readable text rather than raw HTML or fragmented columns.

Second, use a lower learning rate than you would for SFT. CPT modifies the model at a deeper level than instruction tuning, and aggressive learning rates can overwrite useful pre-trained knowledge. Start with 1e-5 and decrease to 5e-6 if you observe quality degradation on general-purpose benchmarks after CPT. The goal is to add domain knowledge without losing the model general capabilities.

Third, validate your CPT model before moving to the SFT stage. Generate responses to domain-specific questions and compare them against the base model. The CPT model should show improved domain vocabulary, more accurate domain-specific reasoning, and better familiarity with domain concepts. If the improvement is not visible, your dataset may be too small, too noisy, or too similar to what the model already learned during pre-training.

Fourth, do not skip the SFT stage after CPT. The CPT model has enhanced domain knowledge but has not been trained to follow instructions. Applying SFT after CPT produces significantly better results than either stage alone. On BIOS, you chain the stages by using the CPT checkpoint as the starting model for your SFT training run.

Continued pre-training begins with the corpus. Collect high-quality text from your domain sources: research papers, technical documentation, internal wikis, textbooks, or curated web crawls. Clean the text by removing boilerplate, deduplicating, and ensuring proper encoding. Format the corpus as a plain text file or JSONL file with one document per line.

Upload the corpus through the BIOS Datasets page. For CPT, the platform accepts plain text files and validates the encoding and formatting. Create a new training run, select Continued Pre-Training as the method, and choose your base model. Set the learning rate to 1e-5, configure the context length to match your document lengths (2048 or 4096 tokens is typical), and set the number of epochs based on your corpus size: 1 to 2 epochs for large corpora, 3 to 5 for smaller ones.

Launch the job and monitor the perplexity metric in the BIOS dashboard. Perplexity should decrease steadily during training, indicating the model is learning the domain language patterns. When perplexity plateaus, the model has absorbed the available signal from your corpus and additional training is unlikely to help. Stop the run at this point to avoid overfitting and wasting compute.

After CPT completes, use the checkpoint as the starting model for an SFT training run. This two-stage pipeline, CPT then SFT, produces domain-specialized models that both understand your domain deeply and follow instructions effectively.