Dataset Preparation for AI Fine-Tuning: Formats and Best Practices

USF BIOS supports four dataset formats: JSONL (JSON Lines), Parquet, CSV, and plain text. JSONL is the most common and recommended format. Each line is a valid JSON object containing the fields for one training example. Parquet is efficient for large datasets due to its columnar compression. CSV works for simple tabular data. Plain text is used for continued pre-training.

All formats go through BIOS validation on upload. The platform checks for structural correctness, required fields, encoding issues, and data type mismatches. If validation fails, you get specific error messages pointing to the problem rows or fields.

The BIOS Datasets page shows all your uploaded datasets with format, size, and status information:

SFT (Supervised Fine-Tuning) datasets contain instruction-response pairs. The standard format uses three fields: instruction (the user query or task description), input (optional additional context), and output (the desired model response). BIOS also supports the messages format used by chat models, where each example is an array of role-content message objects.

Example JSONL line: {"instruction": "Summarize the following article", "input": "The Federal Reserve announced...", "output": "The Fed raised interest rates by 25 basis points..."}

The output field is what the model learns to generate. Make your outputs consistent in style, format, and quality. The model will reproduce the patterns it sees in the training data.

Preference datasets for DPO, SimPO, ORPO, and CPO contain paired responses: prompt, chosen (the preferred response), and rejected (the less preferred response). For KTO, the format is: prompt, response, label (desirable/undesirable).

Example JSONL line: {"prompt": "Explain quantum entanglement", "chosen": "Quantum entanglement is a phenomenon where...", "rejected": "Entanglement is when particles are connected..."}

The quality gap between chosen and rejected responses matters. Pairs where both responses are nearly identical or where the rejected response is actually better than the chosen one will confuse the training signal. Review a sample of your pairs before training.

Quality beats quantity: 5,000 high-quality examples often produce better results than 50,000 noisy ones. Make sure your data is:

• Accurate: outputs are factually correct.
• Consistent: the same format and style across examples.
• Diverse: covering the range of inputs the model will see in production.
• Deduplicated: no repeated examples that would bias the model.

Common quality issues to check for:

• Truncated outputs.
• Inconsistent formatting (mixing markdown and plain text).
• Factual errors in outputs.
• Examples where the output does not actually address the instruction.
• Encoding issues (broken Unicode characters, HTML entities in plain text).

When you upload a dataset to BIOS, the platform runs automatic validation: format checking (valid JSON, correct CSV delimiters), field detection (identifying instruction/output columns), type validation (ensuring fields contain the expected data types), and content validation (checking for empty fields, excessively long examples, and encoding issues).

Here is the BIOS dataset upload interface showing validation results:

The preview feature shows a sample of your processed data before training. You can verify that the instruction-output mapping is correct, check that column mapping is applied correctly, and inspect a few examples for quality. This catches common issues like swapped columns or incorrect field names before you spend compute on training.

The data preview lets you browse individual rows and inspect field values:

BIOS integrates with HuggingFace Hub for dataset import. Connect your HuggingFace token in the Integrations page, then browse and import both public and private datasets directly. The platform handles dataset downloading, format conversion, and column mapping automatically.

When importing from HuggingFace, BIOS detects the dataset format and maps columns to the expected fields. You can adjust the column mapping before confirming the import. This is useful for quickly testing with established benchmark datasets or importing proprietary datasets hosted on your HuggingFace organization.

Connect your HuggingFace account through the Integrations page to enable direct dataset import:

Inconsistent output formatting causes more failed fine-tunes than anything else. If some examples in your SFT dataset produce JSON output while others produce plain text, the fine-tuned model will inconsistently switch between formats. Ensure every example uses the same output format, structure, and style. Run a consistency audit on your dataset before uploading. Check that all outputs follow the same template, use the same delimiters, and adhere to the same length guidelines.

Chasing dataset size is the next trap: too many low-quality examples slip in along the way. A model trained on 50,000 noisy examples will produce noisy outputs. A model trained on 5,000 carefully curated examples will produce clean, consistent outputs. If you are working with a limited annotation budget, invest in quality over quantity. Have annotators revise and improve examples rather than producing more examples at lower quality.

Encoding issues are surprisingly common and easy to miss. Check your dataset for broken Unicode characters, HTML entities in plain text (like & instead of &), inconsistent line endings, and invisible control characters. These artifacts get learned by the model and appear in its outputs. The BIOS validation catches some encoding issues, but not all. Manual inspection of a sample of your data is still important.

For preference datasets specifically, ensure that the preference signal is consistent and meaningful. If different annotators apply different preference criteria (one preferring concise responses while another prefers detailed ones), the training signal is contradictory. Establish clear annotation guidelines, train your annotators, and measure inter-annotator agreement before using the data for training.

BIOS provides a complete dataset management workflow designed to minimize the time between preparing your data and starting a training run. To get started, navigate to the Datasets page in your workspace. You will see any previously uploaded datasets listed with their format, size, and validation status.

To upload a new dataset, click the upload button and select your file. BIOS accepts JSONL, Parquet, CSV, and plain text files up to the workspace storage limit. The platform immediately validates the file format, checks for required fields, and reports any issues. If validation succeeds, you can preview the processed data to verify it looks correct before using it in a training run.

For teams that use HuggingFace Hub, the integration shortens dataset import to a few clicks. Connect your HuggingFace token once through the Integrations page, and then browse both public and your private datasets directly from the BIOS interface. Select a dataset, confirm the column mapping, and import. BIOS handles the download, format conversion, and validation automatically. This is particularly useful for benchmarking against established public datasets or for teams that maintain their proprietary datasets on HuggingFace.

Once your dataset is uploaded and validated, it is available for any training run in your workspace. Multiple training runs can use the same dataset, and datasets persist across sessions. Running comparison experiments with different training methods or hyperparameters on the same data becomes routine, which is essential for rigorous model development.