Fine-Tuning Large Language Models: The Definitive, Step-by-Step Guide (2025)

Welcome to the ultimate guide to fine-tuning Large Language Models (LLMs)! This isn’t just an update; it’s a complete overhaul, incorporating the latest best practices and cutting-edge techniques from 2025. We’ll move beyond basic theory and dive into a practical, code-rich tutorial that empowers you to tailor pre-trained LLMs for your specific needs, even with limited resources.

This tutorial is designed to be your comprehensive roadmap, covering:

Foundational Concepts: A clear understanding of fine-tuning, its benefits, and the landscape of fine-tuning methods.
Model Selection Mastery: Learn how to choose the right base LLM, considering size, architecture, licensing, and more – with concrete model recommendations for 2025.
Data Preparation Excellence: Data is king! We’ll guide you through creating high-quality datasets, formatting them correctly, and leveraging synthetic data generation for specialized tasks.
Hands-on Coding with Transformers: A detailed, runnable Python code example using Hugging Face Transformers, incorporating Parameter-Efficient Fine-Tuning (PEFT) for efficiency.
Evaluation & Tuning Techniques: Master the art of evaluating your fine-tuned model and optimizing its performance through hyperparameter tuning.
Advanced Techniques for 2025: Explore quantization, distributed training, Reinforcement Learning from Human Feedback (RLHF) and its simpler alternatives like GRPO, plus memory-saving strategies like Gradient Checkpointing.
Simplified Fine-tuning with Axolotl: Discover how to streamline your workflow with the user-friendly Axolotl framework.
Deployment Strategies: Learn how to deploy your fine-tuned model for real-world applications.
Troubleshooting & Best Practices (2025 Edition): A robust troubleshooting guide, updated with common pitfalls and solutions for modern fine-tuning.

Let’s embark on this journey to LLM mastery!

Step 1: Demystifying Fine-Tuning: The “Why” and “What”

Before we write a single line of code, let’s solidify our understanding of fine-tuning.

1.1 What Exactly is Fine-Tuning in 2025?

Imagine LLMs as incredibly knowledgeable generalists, pre-trained on vast amounts of data to understand and generate human language. Fine-tuning is the art of transforming these generalists into specialists. It’s a transfer learning technique where we take a pre-trained LLM (from organizations like Meta, Google, Mistral, DeepSeek AI, and more) and further train it on a smaller, task-specific dataset.

Think of it like this: a world-class chef (pre-trained LLM) knows general cooking techniques. Fine-tuning is like specializing that chef in French pastry (your specific task) by having them practice extensively on French pastry recipes and techniques (your smaller, specific dataset). The chef retains their general culinary skills but becomes an expert in French pastry.

Fine-tuning adjusts the internal weights of the pre-trained model. These weights are the model’s knowledge representation. By training on your specific data, you’re subtly nudging these weights to better represent the nuances and patterns of your target task, while still leveraging the broad language understanding learned during pre-training.
1.2 Why Fine-Tune in the Age of Powerful Base Models? (The Compelling Advantages)

In 2025, base LLMs are incredibly powerful. So, why bother fine-tuning? Here’s why it’s still essential:
- Unmatched Performance Boost: Fine-tuned models consistently outperform general-purpose LLMs on specific tasks. You’ll achieve higher accuracy, better fluency, and more relevant outputs tailored to your domain. Think of the DeepSeek-R1 example – fine-tuning turns it into a Python coding expert!
- Data Efficiency – Your Data Goes Further: Training LLMs from scratch requires petabytes of data and massive compute. Fine-tuning is incredibly data-efficient. Thousands, or even hundreds, of high-quality examples can be sufficient to achieve significant improvements. This is crucial when you have limited, specialized data.
- Resource Efficiency – Accessible to More: Fine-tuning drastically reduces computational demands. You don’t need a supercomputer. Many fine-tuning tasks, especially with Parameter-Efficient Fine-tuning (PEFT), can be accomplished on consumer-grade GPUs or cost-effective cloud instances. Unsloth and other libraries are pushing this efficiency even further!
- Customization & Control – Own Your AI: Fine-tuning allows you to imbue your model with a specific style, tone, knowledge base, and even personality. You control the output. No more relying solely on the unpredictable outputs of a general model. This is key for brand voice, specialized domains, and ensuring consistent behavior.
- Adaptability & Iteration – Stay Agile: The AI landscape is constantly evolving. Fine-tuning allows you to quickly adapt your model to new tasks, changing data, or refined requirements. Need your chatbot to handle a new product line? Fine-tune it on data related to that product.
- Bias Mitigation & Alignment: Pre-trained models can inherit biases from their vast training data. Fine-tuning gives you a chance to steer the model towards less biased and more aligned behavior for your specific use case by carefully curating your fine-tuning dataset.
1.3 Navigating the Fine-Tuning Landscape: Types & Techniques in 2025

The fine-tuning world is rich and varied. Here’s a breakdown of the key approaches you’ll encounter:
- Full Fine-Tuning (The Classic, but Resource-Intensive): This is the traditional method where all the parameters of the pre-trained model are updated during training. While it offers the highest potential for adaptation, it’s computationally expensive and can lead to catastrophic forgetting, where the model loses its general language abilities. Generally less favored in 2025 due to PEFT advancements.
- Parameter-Efficient Fine-Tuning (PEFT) – The Smart Choice for 2025: PEFT techniques are revolutionizing fine-tuning. They update only a small fraction of the model’s parameters, achieving comparable performance to full fine-tuning with dramatically reduced computational cost and risk of catastrophic forgetting. PEFT is highly recommended for most practical scenarios. Key PEFT methods include:
  - LoRA (Low-Rank Adaptation) – The Industry Standard: LoRA is incredibly popular for its effectiveness and efficiency. It adds small, trainable “adapter” layers to the existing model architecture. Crucially, the original pre-trained weights are frozen, preserving the model’s general language knowledge. LoRA significantly reduces the number of trainable parameters, making fine-tuning faster and requiring less memory. The Unsloth library specializes in optimized LoRA implementation for even greater speed and memory savings.
  - Prefix Tuning & Prompt Tuning: These methods add trainable vectors (prefixes or soft prompts) to the input embeddings. They are less widely used than LoRA but can be effective in certain scenarios.
  - IA³ (Infused Adapter by Inhibiting and Amplifying Inner Activations): IA³ learns scaling factors for the model’s internal activations. Another PEFT method, offering different trade-offs.
- Instruction Fine-Tuning – Teaching Models to Follow Orders: This is essential for creating conversational models and assistants. You train the model on a dataset of instructions and their desired outputs. This teaches the model to understand and follow instructions, generalizing to new, unseen instructions. The data format is typically structured as conversations, like this JSON Lines example:
```
{"conversations": [{"role": "system", "content": "You are a helpful, concise assistant."}, {"role": "user", "content": "Explain the theory of relativity in simple terms."}, {"role": "assistant", "content": "Einstein's theory of relativity explains how time and space are relative and interconnected.  Basically, gravity isn't a force, but a curve in spacetime caused by mass and energy."}]}
{"conversations": [{"role": "user", "content": "Write a short poem about autumn leaves."}, {"role": "assistant", "content": "Crimson and gold, they gently fall,\nA whispered hush, embracing all.\nAutumn's embrace, a fleeting grace,\nNature's beauty, in time and space."}]}
```
- Reinforcement Learning from Human Feedback (RLHF) – Aligning with Human Preferences (and its Simpler Alternatives): RLHF is a more advanced, multi-stage technique for aligning models with human values like helpfulness, harmlessness, and truthfulness. It’s complex but can yield significant improvements in output quality and alignment. The traditional RLHF pipeline involves:
  1. Supervised Fine-Tuning (SFT): Initial fine-tuning on instruction-output pairs to get the model started.
  2. Reward Model Training: A separate model is trained to predict human preferences. Humans rank different model outputs for the same prompt, and the reward model learns to mimic these rankings.
  3. Reinforcement Learning (RL) Fine-tuning: The LLM is then fine-tuned using reinforcement learning algorithms (like PPO) to maximize the rewards predicted by the reward model.
  - DPO (Direct Preference Optimization) – A Streamlined RLHF: DPO simplifies RLHF by directly optimizing the LLM based on preference data, without explicitly training a separate reward model. It’s more stable and computationally efficient than traditional RLHF.
  - ORPO (Odds Ratio Preference Optimization) & GRPO (Group Relative Policy Optimization): Emerging alternatives to standard RLHF and DPO, offering potentially improved stability and performance. GRPO, in particular, is gaining traction for its simplicity and speed, making RL-based fine-tuning more accessible, even for smaller models (as demonstrated in the DataWizz AI article using Qwen-0.5B!). GRPO focuses on directly optimizing for high-reward trajectories, simplifying the RL process and making it remarkably efficient.

Step 2: Choosing Your Base Model: A 2025 Guide to Selection

Selecting the right pre-trained LLM is a critical decision. It sets the foundation for your fine-tuning success. Consider these factors carefully:

2.1 Model Size – Balancing Power and Resources:

Model size, measured by the number of parameters, directly impacts performance and resource requirements. In 2025, you have a wide range of sizes to choose from:
- Small (1.5B - 7B parameters): Highly accessible. Fine-tuning is feasible on consumer-grade GPUs (NVIDIA RTX 3090, 4090) or smaller cloud instances like Google Colab. Ideal for experimentation, resource-constrained projects, and tasks where extreme scale isn’t necessary. Excellent Examples for 2025: Qwen2.5-1.5B-Instruct, Llama-3.2-3B-Instruct, Gemma-2-2b-it, Phi-3-mini-instruct. Quantized versions (e.g., unsloth/gemma-2-2b-it-bnb-4bit) further reduce resource needs, thanks to libraries like Unsloth and BitsAndBytes.
- Medium (14B - 32B parameters): A sweet spot balancing performance and resource demands. Require more powerful GPUs (NVIDIA A100, H100) or multi-GPU setups. Good for more complex tasks where higher performance is needed without the extreme resource footprint of massive models. Strong Choices for 2025: Coma-II-14B, Arcee-Blitz (24B), TheBeagle-v2beta-32B-MGS, QwQ-32B Kumo.
- Large (70B+ parameters): State-of-the-art performance for the most demanding tasks. Require significant infrastructure – multiple high-end GPUs (A100s, H100s) or cloud-based GPU clusters. Best for pushing the boundaries of LLM capabilities when resources aren’t a primary constraint. Top-Tier Models in 2025: Qwen2.5-72B-Instruct, Meta-Llama-3.1-70B-Instruct, AceInstruct-72B, Tulu 3 405B (for research at scale).
2.2 Pre-training Data – Shaping the Model’s Foundation:

The data a model is pre-trained on profoundly influences its capabilities, biases, and suitability for different tasks.
- General-Purpose Models: Trained on massive, diverse datasets (web text, books, code, etc.). Versatile and suitable for a wide range of tasks. Examples: Qwen2.5, Llama 3, Mistral 7B, Gemma 2.
- Specialized Models: Pre-trained on specific domains (scientific papers, code repositories, legal documents). May offer an advantage if your fine-tuning task aligns with the pre-training domain. Examples: EpistemeAI/Athena-gemma-2-2b-it-Philos (reasoning, philosophy), DeepSeek-R1-Distill-Llama-3B (reasoning), RecurvAI/Recurv-Medical-Deepseek-R1 (medical domain). For instance, if you’re building a legal AI, a model pre-trained on legal text might be a better starting point than a general-purpose model.
2.3 Model Architecture – Decoder-Only Dominance in 2025:

Almost all modern LLMs are based on the Transformer architecture. The dominant architecture for text generation in 2025 is:
- Causal Language Models (Decoder-Only): These models are trained to predict the next token in a sequence. They are ideal for text generation tasks like chatbots, creative writing, code generation, and instruction following. Examples: Qwen, Llama, Mistral, Gemma, Phi. This tutorial primarily focuses on decoder-only models, as they are the most relevant for fine-tuning for generative tasks.
- Encoder-Decoder Models: Models like T5 and BART have separate encoders and decoders. They are often used for tasks like translation and summarization. While fine-tunable, they are less common for general-purpose text generation compared to decoder-only models.
2.4 Licensing – The Legal Foundation for Your Project (CRITICAL!):

Licensing is non-negotiable. It dictates how you can legally use the model, especially for commercial purposes. Always verify the license before you choose a model.
- Permissive Licenses (Apache 2.0, MIT, Llama 3 License, etc.): Offer broad usage rights, typically including commercial use, modification, and distribution. These licenses are generally preferred for maximum flexibility.
- Non-Commercial Licenses: Strictly limit use to research, personal projects, or non-profit purposes. Commercial applications are prohibited. Example: AceInstruct models have non-commercial licenses. Using these for commercial purposes could lead to legal issues.
- Custom Licenses: Some models may have unique licenses. Read them carefully to understand the terms and conditions.
2.5 Existing Fine-tuning – Leverage Pre-Existing Specialization:

Many models available on platforms like Hugging Face are already fine-tuned for specific purposes.
- “Instruct” Models: Fine-tuned on instruction-following datasets. Excellent starting points for tasks involving instructions, question answering, and general conversational AI. Examples: Qwen2.5-72B-Instruct, Meta-Llama-3.1-70B-Instruct, Phi-3-mini-instruct. These are often the best choice for general-purpose fine-tuning.
- Chat Models: Specifically fine-tuned for conversational interactions. Optimized for dialogue and multi-turn conversations.
- Task-Specific Fine-tunes: Models fine-tuned for coding, math, medical text, etc. If your task is highly specialized, explore models already fine-tuned in that domain.
2.6 Language Support – Multilingual Capabilities in a Global World:

If your application needs to handle multiple languages, choose a multilingual base model. Models like Qwen2.5, EXAONE 3.5, and AGO solutions Llama-3.1-SauerkrautLM-70b-Instruct are explicitly designed for multilingual performance.
2.7 Prompt Template – The Secret Language of LLMs (ABSOLUTELY CRITICAL!):

Different LLMs expect different input formats. Using the wrong prompt template is the single most common cause of poor fine-tuning results and frustration. It’s like speaking the wrong language to the model. You must use the correct template for your chosen base model.
- ChatML Template: Widely used by Qwen models and many chat-optimized models. Uses special tokens like <|im_start|> and <|im_end|>.
- Llama 3 Template: Specific to Meta Llama 3 models. Employs tokens like <|start_header_id|>, <|end_header_id|>, and <|eot_id|>.
- Gemma Template: Used by Google’s Gemma family.
- Phi-3 Template: Uses <|user|> and <|assistant|> tags.
- DeepSeek Template: Can involve custom tags like <think> </think> for reasoning models.
- Custom Templates: Some models have unique, proprietary formats.
Where to find the correct template:
- Model Card on Hugging Face Hub: Always check the model card. It should specify the required prompt template, often with example code.
- Model Documentation: The model’s official documentation (if available) is another reliable source.
- Hugging Face transformers Library – Your Template Translator: The transformers library provides the tokenizer.apply_chat_template() function. This powerful function automatically handles the complexities of prompt formatting for many models. Use this function! It will save you countless headaches and ensure your prompts are correctly formatted. We’ll demonstrate this in the code examples.
2.8 Uncensored/Abliterated Models – Proceed with Extreme Caution:

Models like huihui-ai/Qwen2.5-72B-Instruct-abliterated have had safety filters *intentionally removed. *
- Dangers: These models can generate highly offensive, biased, harmful, or inappropriate content. They lack the safety guardrails of standard models.
- Use Cases: Extremely niche and specialized. Only consider these if you have a very specific, justified need for uncensored output and fully understand the ethical and potential legal risks. Generally, for most applications, using censored, safety-aligned models is strongly recommended.
2.9 Merged Models – Experimental Power (and Potential Instability):

Models created using merging tools like mergekit (e.g., TheBeagle-v2beta-32B-MGS, QwQ-32B Kumo, Rombos-LLM-V2.5-Qwen-32b) are combinations of multiple models.
- Potential Benefits: Can inherit strengths from multiple base models, potentially leading to enhanced performance in specific areas.
- Risks: Merged models are often experimental. Their behavior can be less predictable, and they may exhibit unexpected issues or biases. Use with caution and thorough evaluation.

Step 3: Data Preparation: The Foundation of Fine-Tuning Success

Data preparation is the most critical step in the fine-tuning process. Garbage in, garbage out. The quality, relevance, and format of your data directly determine the performance of your fine-tuned model. Invest time and effort here!

3.1 Dataset Format – JSON Lines (.jsonl) – The Gold Standard:
- JSON Lines (.jsonl): The recommended and widely adopted format for fine-tuning datasets. Each line in the file is a valid JSON object, representing a single training example.
- Advantages:
  - Readability & Simplicity: Easy to read, write, and parse.
  - Streaming Efficiency: Large datasets can be processed line by line, without loading the entire dataset into memory.
  - Hugging Face Datasets Compatibility: Seamlessly integrates with the Hugging Face datasets library.
3.2 Dataset Structure – Tailor it to Your Task:

The structure of your JSON objects within the .jsonl file depends on the fine-tuning task. Here are common examples:
- Instruction Following/Chat (Conversational Models): (Most prevalent for modern LLMs)
```
{"conversations": [{"role": "system", "content": "You are a helpful, creative writer."}, {"role": "user", "content": "Write a short story about a cat who dreams of flying."}, {"role": "assistant", "content": "Whiskers, a tabby of discerning taste, dreamt not of tuna, but of flight.  He'd leap from rooftops, paws outstretched, imagining feathered wings instead of fur. One night, soaring through a moonlit sky in his dream, he felt the wind beneath him, truly flying. He awoke with a jolt, a phantom breeze still tingling his whiskers, the taste of freedom lingering on his tongue."}]}
{"conversations": [{"role": "user", "content": "Translate 'Thank you very much' into Spanish."}, {"role": "assistant", "content": "Muchas gracias"}]}
```
  - conversations: A list of conversation turns (messages).
  - role: Specifies the speaker: "system" (initial context, optional but highly recommended), "user" (user input/instruction), "assistant" (model’s desired response).
  - content: The actual text of the message.
  - Key Rule: Conversations should alternate between "user" and "assistant". The "system" message, if present, should always be the first message.
- Text Completion (Basic Generation):
```
{"input": "The weather today is", "output": " sunny and warm, with a gentle breeze."}
{"input": "Write a haiku about rain:\n", "output": "Soft drops on the pane,\nNature's gentle, cleansing tears,\nEarth drinks, life reborn."}
```
  - input: The starting text prompt.
  - output: The desired completion/continuation.
- Adapt for Other Tasks: You can customize the structure for tasks like:
  - Summarization: {"input": "Long document text...", "output": "Concise summary..."}
  - Translation: {"input": "Source language text", "target_language": "fr", "output": "Target language translation"}
  - Question Answering: {"context": "Background information...", "question": "User question?", "answer": "The answer."}

3.3 Prompt Formatting – Absolute Precision is Key (Repeating for Emphasis!)

We cannot stress this enough: Using the correct prompt template is absolutely essential for successful fine-tuning. Incorrect formatting will lead to subpar performance, regardless of data quality or model choice.

ChatML Example (for models like Qwen):

<|im_start|>system
You are a helpful and concise assistant.<|im_end|>
<|im_start|>user
What are the benefits of meditation?<|im_end|>
<|im_start|>assistant
Meditation reduces stress, improves focus, and promotes emotional well-being.<|im_end|>

Llama 3 Template Example:

<|start_header_id|>system<|end_header_id|>
You are a helpful and concise assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
What are the benefits of meditation?<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>
Meditation reduces stress, improves focus, and promotes emotional well-being.<|eot_id|>

Phi-3 Template Example:

<|user|>What is the capital of France?<|assistant|>Paris

DeepSeek Custom Prompt Example (Reasoning Model):

<|start_header_id|>system<|end_header_id|>
Respond in the following format:
<think>
You should reason between these tags.
</think>

Answer goes here...

Always use <think> </think> tags even if they are not necessary.
<|eot_id|>

<|start_header_id|>user<|end_header_id|>
Which one is larger? 9.11 or 9.9?<|eot_id|>

<|start_header_id|>assistant<|end_header_id|>
<think>
9.11 and 9.9.
9.11 - 9 = 0.11.
9.9 - 9 = 0.9.
0.9 > 0.11.
So 9.9 > 9.11.
Actually 0.11 < 0.9.
So 9.11 < 9.9.
9.9 is larger.
</think>
9.9
<|eot_id|>

Leveraging tokenizer.apply_chat_template() – The Smart Way:

from transformers import AutoTokenizer

# ChatML example (e.g., Qwen)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-7B-Instruct")
messages = [{"role": "user", "content": "What is the capital of France?"}]
formatted_prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(formatted_prompt)

# Llama 3 example
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
messages = [{"role": "user", "content": "What is the capital of France?"}]
formatted_prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(formatted_prompt)

# Phi-3 Example
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-instruct")
messages = [{"role": "user", "content": "What is the capital of France?"}]
formatted_prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(formatted_prompt)

Best Practice: Always use tokenizer.apply_chat_template() to format your prompts. It handles the intricacies of special tokens, system messages, and model-specific formatting automatically, minimizing errors and ensuring compatibility.

3.4 Data Quality – The Cornerstone of Performance:
- Accuracy: Ensure factual correctness in your data, especially for knowledge-intensive tasks.
- Consistency: Maintain a consistent style, tone, and formatting throughout your dataset.
- Relevance: Data must be directly relevant to your target task. Focus on examples that exemplify the desired behavior of your fine-tuned model.
- Diversity: Include a wide range of examples covering different scenarios, input types, and desired outputs. Diversity helps the model generalize better.
- Bias Awareness & Mitigation: Be mindful of potential biases in your data. Data can inadvertently perpetuate societal biases. Strive to create a dataset that is as fair and unbiased as possible, or at least understand the biases your data might introduce. Consider techniques to mitigate bias during data collection and preprocessing.
3.5 Data Quantity – Quality Over Quantity (But Both Matter):
- Quality > Quantity: A smaller dataset of high-quality, relevant examples is far more effective than a massive dataset of noisy, irrelevant data.
- Thousands are Often Sufficient: For many fine-tuning tasks, a few thousand (or even a few hundred for highly specialized tasks) carefully curated examples can yield excellent results.
- Data Augmentation: If you have limited data, explore data augmentation techniques to artificially increase the size and diversity of your dataset (e.g., paraphrasing, back-translation, synonym replacement).
- Synthetic Data Generation – A Powerful Tool for 2025: As highlighted in the DeepSeek-R1 article, synthetic data generation is becoming increasingly important, especially for specialized tasks or data-scarce domains. Use LLMs themselves (like DeepSeek-R1-Distill-Qwen-32B or Llama 3) to generate synthetic data that aligns with your target task. Tools like the Synthetic Data Generator (mentioned in the article) simplify this process. For example, you can generate synthetic reasoning datasets, coding problem datasets, or datasets for specific industries.
3.6 Data Preprocessing – Preparing Data for the Model’s Consumption:
- Cleaning: Remove noise and irrelevant information: HTML tags, special characters, excessive whitespace, etc. Standardize text encoding (UTF-8).
- Tokenization (Handled by Tokenizer): The tokenizer (loaded from your pre-trained model) converts text into numerical tokens that the model understands. You don’t usually need to do this manually, but be aware that the tokenizer is a crucial component. tokenizer.apply_chat_template() and the Trainer class automatically handle tokenization for you.
- Truncation & Padding – Managing Sequence Length: LLMs have a maximum context length (e.g., 2048, 4096, 32768 tokens). You must handle input and output sequences that exceed this limit.
  - Truncation: Shorten sequences to fit within the maximum length. Be careful to truncate intelligently, ideally from the beginning of the input or output, or use techniques that preserve key information (if possible).
  - Padding: Add padding tokens (special tokens like <pad>) to shorter sequences to make all sequences in a batch the same length. This is essential for efficient batch processing during training. The transformers library and DataCollatorForLanguageModeling handle padding automatically.
- Data Splitting – Train, Validate, Test: Divide your dataset into three crucial sets:
  - Training Set (Largest): Used to train the model’s parameters. Aim for 70-80% of your data here.
  - Validation Set (Development Set): Used during training to monitor performance, tune hyperparameters, and prevent overfitting. Typically 10-15% of your data. The model does not train directly on the validation set; it’s used for evaluation during training.
  - Test Set (Hold-Out Set): Used for the final, unbiased evaluation of your fine-tuned model’s performance after training is complete. This set should be completely separate from the training and validation data. Typically 10-15% of your data.

Step 4: Setting Up Your Fine-Tuning Environment – Hardware & Software

Let’s get your environment ready for fine-tuning.

4.1 Hardware – GPUs are Your Best Friend:
- GPU (Graphics Processing Unit): Fine-tuning LLMs almost always requires a GPU. GPUs provide the parallel processing power necessary for efficient deep learning. VRAM (GPU memory) is the key constraint. VRAM needs depend on:
  - Model Size: Larger models require more VRAM.
  - Batch Size: Larger batch sizes increase VRAM usage.
  - Sequence Length: Longer input/output sequences consume more VRAM.
  - Data Type (Precision): Lower precision (e.g., 4-bit quantization) reduces VRAM.
- VRAM Guidelines (Approximate):
  - Small Models (1.5B - 7B): 8GB-24GB VRAM (NVIDIA RTX 3060, RTX 3090, RTX 4090, etc.). Consumer GPUs are often sufficient.
  - Medium Models (14B - 32B): 24GB-80GB+ VRAM (NVIDIA A100, H100, or multiple GPUs). High-end GPUs or multi-GPU setups are needed.
  - Large Models (70B+): Multiple high-end GPUs (A100s, H100s) with 80GB+ VRAM each. Cloud-based GPU clusters are usually necessary.
- Cloud-Based GPUs – Accessible Compute Power: If you lack a suitable local GPU, cloud platforms offer on-demand GPU instances:
  - Google Colab: Free tier with limited GPUs (T4, sometimes P100), paid Colab Pro/Pro+ with more powerful GPUs (A100, V100). Excellent for experimentation and smaller models.
  - Amazon SageMaker: AWS’s machine learning platform, offering a wide range of GPU instances (including A100, H100).
  - Google Cloud AI Platform (Vertex AI): Google’s cloud ML platform, similar to SageMaker.
  - Microsoft Azure Machine Learning: Azure’s ML service, with GPU options.
  - Paperspace Gradient: Cloud GPUs specifically for ML, competitive pricing, and user-friendly interface.
  - RunPod: Another cloud GPU provider, often with more affordable options.
4.2 Software – Essential Libraries for LLM Fine-tuning:

Install these Python libraries using pip:
- Python: Version 3.8 or later is recommended.
- PyTorch: The dominant deep learning framework for LLMs. Install the correct version for your CUDA setup if using NVIDIA GPUs. Follow the instructions on https://pytorch.org/.
- Hugging Face Transformers: pip install transformers. The core library for LLMs, providing models, tokenizers, training utilities, and much more.
- Hugging Face Accelerate: pip install accelerate. Simplifies distributed training across multiple GPUs and machines.
- Hugging Face Datasets: pip install datasets. For efficient loading and processing of datasets, including .jsonl files and Hugging Face datasets.
- Optional but Highly Recommended Libraries (Install as needed):
  - PEFT (Parameter-Efficient Fine-Tuning): pip install peft. For LoRA and other PEFT techniques. Essential for efficient fine-tuning.
  - BitsAndBytes (Quantization): pip install bitsandbytes. For 8-bit and 4-bit quantization to reduce memory usage.
  - Unsloth: (Optimized Fine-tuning) Install from GitHub following their instructions: https://github.com/unslothai/unsloth. Unsloth provides significant speed and memory optimizations for fine-tuning, especially with LoRA. It’s highly recommended, as seen in the provided articles.
  - TRL (Transformer Reinforcement Learning): pip install trl. For RLHF and GRPO (using the Hugging Face GRPO trainer).
  - DeepSpeed: (For Extremely Large Models and Distributed Training) Install from GitHub: https://github.com/microsoft/DeepSpeed. For advanced distributed training and memory optimization, primarily for massive models.
  - Axolotl: (Simplified Fine-tuning Framework) Install following instructions on their GitHub repo: https://github.com/OpenAccess-AI-Collective/axolotl. Axolotl simplifies fine-tuning configuration using YAML files.
  - Mergekit: (Model Merging) pip install mergekit. If you want to experiment with merging models.
  - Flash Attention 2: For faster and more memory-efficient attention mechanism if your GPU and software setup supports it. pip install flash-attn --no-build-isolation (installation can be complex, refer to Flash Attention documentation).

Step 5: Writing Your Fine-Tuning Code: Hands-on with Hugging Face Transformers

Let’s put theory into practice with a complete, runnable Python code example using the Hugging Face Trainer class. This example incorporates best practices and PEFT (LoRA) for efficient fine-tuning.

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments,
    Trainer,
    DataCollatorForLanguageModeling,
)
from datasets import load_dataset

# --- 1. Configuration: Define your fine-tuning parameters here ---
MODEL_NAME = "Qwen/Qwen1.5-1.8B-Chat"  # **[REQUIRED]** Replace with your chosen base model
DATASET_PATH = "your_dataset.jsonl"  # **[REQUIRED]** Path to your .jsonl dataset file
OUTPUT_DIR = "fine_tuned_model"
PER_DEVICE_TRAIN_BATCH_SIZE = 2  # Adjust based on your GPU memory. Start small (e.g., 1, 2) and increase if possible. Gemma 2B models often use batch size of 2 or 4 on T4.
GRADIENT_ACCUMULATION_STEPS = 8  # Simulate larger batch sizes. Increase if you have limited GPU memory. Effective batch size = PER_DEVICE_TRAIN_BATCH_SIZE * GRADIENT_ACCUMULATION_STEPS. Common value from examples: 8.
LEARNING_RATE = 3e-5  # **[Improved Value]** A slightly higher learning rate, common for LoRA fine-tuning and Gemma models. Range: 2e-5 to 5e-5 is generally good.
NUM_TRAIN_EPOCHS = 3  # Number of passes through your training data.  1-3 epochs are often sufficient for fine-tuning with LoRA.
MAX_SEQ_LENGTH = 2048  # **[Improved Value]** Increased max sequence length to 2048, common for modern models and datasets. Adjust based on your model and data.
USE_PEFT = True  # **[RECOMMENDED]** Parameter-Efficient Fine-Tuning (LoRA).
LORA_R = 32  # **[Improved Value]**  Higher LoRA rank (32) for potentially better performance, especially for larger models like Gemma 2B. Common range: 16-32.
LORA_ALPHA = 64  # **[Improved Value]** LoRA alpha often set to double the rank.
LORA_DROPOUT = 0.0  # **[Improved Value]** Dropout often set to 0 in LoRA for fine-tuning for optimized performance.

# --- 2. Load Model and Tokenizer ---
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
    quantization_bit=4, # **[Improved Value]** Load model in 4-bit for memory efficiency, as seen in Gemma examples.
    load_in_4bit=True, # Explicitly enable 4-bit loading for Unsloth/bitsandbytes models.
)

# --- 3. Prepare PEFT (Parameter-Efficient Fine-Tuning) if enabled ---
if USE_PEFT:
    from peft import LoraConfig, get_peft_model

    peft_config = LoraConfig(
        r=LORA_R,
        lora_alpha=LORA_ALPHA,
        lora_dropout=LORA_DROPOUT,
        bias="none",
        task_type="CAUSAL_LM",
        target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"] # Target all linear layers as suggested by some examples and for broader adaptation.
    )
    model = get_peft_model(model, peft_config)
    model.print_trainable_parameters()

# --- 4. Load and Prepare Dataset ---
dataset = load_dataset("json", data_files=DATASET_PATH, split="train")

def preprocess_function(examples):
    formatted_conversations = [
        tokenizer.apply_chat_template(example["conversations"], tokenize=False, add_generation_prompt=True)
        for example in examples["conversations"]
    ]

    model_inputs = tokenizer(
        formatted_conversations,
        max_length=MAX_SEQ_LENGTH,
        padding="max_length",
        truncation=True,
        return_tensors="pt",
        return_attention_mask=True,
    )
    model_inputs["labels"] = model_inputs["input_ids"].clone()

    return model_inputs

processed_dataset = dataset.map(
    preprocess_function,
    batched=True,
    remove_columns=dataset.column_names,
    num_proc=4,
)

processed_dataset = processed_dataset.train_test_split(test_size=0.1, seed=42)
train_dataset = processed_dataset["train"]
eval_dataset = processed_dataset["test"]

# --- 5. Training Arguments ---
training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,
    per_device_train_batch_size=PER_DEVICE_TRAIN_BATCH_SIZE,
    gradient_accumulation_steps=GRADIENT_ACCUMULATION_STEPS,
    learning_rate=LEARNING_RATE,
    num_train_epochs=NUM_TRAIN_EPOCHS,
    logging_dir=f"{OUTPUT_DIR}/logs",
    logging_steps=5,  # **[Improved Value]** Log more frequently, every 5 steps for better monitoring.
    evaluation_strategy="steps",
    eval_steps=100,
    save_strategy="steps",
    save_steps=100, # **[Improved Value]** Save checkpoints more frequently, every 100 steps for better iteration.
    save_total_limit=2,
    fp16=False,
    bf16=True, # Use bfloat16 for modern GPUs.
    report_to="tensorboard",
    optim="adamw_8bit", # AdamW 8-bit optimizer for memory efficiency.
    lr_scheduler_type="cosine", # Cosine learning rate scheduler, common and effective.
    warmup_ratio=0.1, # Warmup ratio of 10%, common in fine-tuning.
    weight_decay=0.01, # **[Improved Value]** Added weight decay for regularization, value 0.01 is common.
    gradient_checkpointing=True,
    dataloader_num_workers=4,
    max_grad_norm=1.0,
    group_by_length=True, # **[Improved Value]** Group by length for potential efficiency gains, especially with mixed sequence lengths.
    # flash_attention="auto", # Enable Flash Attention 2 if compatible - may require specific hardware and software setup.
)

# --- 6. Data Collator ---
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

# --- 7. Trainer Initialization ---
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    data_collator=data_collator,
    tokenizer=tokenizer,
)

# --- 8. Train the Model ---
print("*** Starting Training ***")
train_result = trainer.train()
trainer.save_model()
tokenizer.save_pretrained(OUTPUT_DIR)
training_metrics = train_result.metrics
trainer.log_metrics("train", training_metrics)
trainer.save_metrics("train", training_metrics)
trainer.save_state()

# --- 9. Evaluation (Optional, but highly recommended) ---
print("*** Starting Evaluation ***")
eval_metrics = trainer.evaluate()
trainer.log_metrics("eval", eval_metrics)
trainer.save_metrics("eval", eval_metrics)

print(f"Fine-tuning complete! Model saved to {OUTPUT_DIR}")
print(f"Training metrics: {training_metrics}")
print(f"Evaluation metrics: {eval_metrics}")

How to Run This Code (Step-by-Step):

Environment Setup: Ensure you have installed all the required libraries (Step 4.2). Activate your Python environment.
Dataset Preparation: Create your .jsonl dataset file (Step 3). Verify that it’s correctly formatted and placed at the DATASET_PATH specified in the configuration.
Configuration – Customize for Your Task:
- MODEL_NAME: [REQUIRED] Replace "Qwen/Qwen1.5-1.8B-Chat" with the Hugging Face model identifier of your chosen base model (Step 2). Examples: "meta-llama/Meta-Llama-3-8B-Instruct", "microsoft/Phi-3-mini-instruct".
- DATASET_PATH: [REQUIRED] Set this to the correct path to your .jsonl dataset file.
- OUTPUT_DIR: Choose a directory to save your fine-tuned model.
- Hyperparameters: Adjust the parameters in the Configuration section (batch size, learning rate, epochs, sequence length, LoRA parameters, etc.) based on your hardware, dataset size, and task complexity. Start with the defaults and experiment (Step 7).
- target_modules (LoRA): [CRITICAL] If using LoRA (USE_PEFT=True), carefully verify and adjust the target_modules list in the LoraConfig. The provided list is a common starting point but may need to be modified for different model architectures. Consult the model’s documentation or source code.
Run the Script: Execute the Python script. Training will commence, and progress will be displayed in your console. Training metrics, checkpoints, and logs will be saved to the OUTPUT_DIR. You can monitor progress using TensorBoard by running tensorboard --logdir fine_tuned_model/logs in your terminal and opening the TensorBoard URL in your browser.

Step 6: Evaluating Your Fine-Tuned Model – Beyond Loss Curves

After fine-tuning, it’s crucial to rigorously evaluate your model’s performance. Don’t rely solely on training loss – that only tells you how well the model learned to fit the training data, not how well it generalizes or performs on your target task.

Quantitative Evaluation – Metrics that Measure:
- Open LLM Leaderboard Benchmarks: Submit your model to the Open LLM Leaderboard to compare its performance against other models on standardized benchmarks. This provides objective metrics and allows the community to understand your model’s capabilities.
- Perplexity: A common metric for language models, indicating how well the model predicts the next token in a sequence. Lower perplexity is generally better. The Trainer class automatically calculates perplexity on your evaluation set (if provided). However, perplexity alone is not sufficient to judge the quality of generated text for specific tasks.
- Task-Specific Metrics – The Real Measure of Success: Use metrics that directly assess performance on your intended task. Examples:
  - Question Answering: ROUGE-L, BLEU, Exact Match (EM), F1-score. ROUGE-L (as used in the DataWizz AI article) is a robust metric for evaluating text similarity and overlap.
  - Summarization: ROUGE (ROUGE-1, ROUGE-2, ROUGE-L).
  - Translation: BLEU (Bilingual Evaluation Understudy), METEOR.
  - Classification (e.g., Sentiment Analysis): Accuracy, Precision, Recall, F1-score.
- Integrating Metrics into Training (Advanced): You can customize the Trainer to calculate and report task-specific metrics during training by defining a compute_metrics function and passing it to the Trainer. See the Hugging Face Trainer documentation for details.
Qualitative Evaluation – Human Judgment is Essential:
- Manual Inspection – Your Eyes and Brain are Powerful Tools: Critically review text generated by your fine-tuned model. Generate outputs for a variety of prompts and assess:
  - Fluency and Coherence: Does the text flow naturally? Is it grammatically correct? Is it easy to understand?
  - Relevance: Does the output directly address the prompt or instruction?
  - Correctness/Accuracy: Is the information presented factually accurate (if applicable)?
  - Style and Tone: Does the model generate text in the desired style and tone?
  - Helpfulness and Harmlessness: Is the model generating helpful and harmless responses? (Especially important for conversational models).
- Human Evaluation (If Feasible) – Crowdsourced or Expert Review: For more rigorous evaluation, especially for production models, consider human evaluation:
  - Crowdsourcing: Use platforms like Amazon Mechanical Turk or Prolific to have multiple human evaluators rate model outputs based on specific criteria (helpfulness, relevance, fluency, etc.).
  - Expert Evaluation: For specialized tasks (e.g., medical or legal AI), have domain experts evaluate the model’s outputs.
- Example of Text Generation for Qualitative Assessment:
```
from transformers import pipeline

# Load your fine-tuned model and tokenizer
generator = pipeline("text-generation", model=OUTPUT_DIR, tokenizer=OUTPUT_DIR, device=0)  # Use device=0 for GPU inference

# Generate text with a prompt (adjust prompt template as needed)
prompt = "<|im_start|>user\nExplain the concept of quantum entanglement in simple terms.<|im_end|>\n<|im_start|>assistant\n"  # Use appropriate prompt format for your model (e.g., ChatML, Llama 3 template)
generated_text = generator(prompt, max_length=100, num_return_sequences=1, temperature=0.7) # Adjust generation parameters (max_length, temperature, etc.)
print(generated_text[0]['generated_text'])
```

Step 7: Hyperparameter Tuning – Optimizing Performance Through Experimentation

Hyperparameter tuning is the process of finding the best combination of training settings (hyperparameters) to maximize your model’s performance. It’s an iterative process of experimentation and evaluation.

Key Hyperparameters to Tune:
- Learning Rate: Controls the step size during weight updates.
  - Typical Range: 1e-5 to 5e-5 is a good starting point.
  - Too High: Training instability, loss spikes, model may not converge.
  - Too Low: Slow training, model may get stuck in local optima.
- Batch Size (per_device_train_batch_size, gradient_accumulation_steps): Affects training speed and memory usage.
  - Larger Batch Size (if memory allows): Can speed up training.
  - Smaller Batch Size: Required if you have limited GPU memory. gradient_accumulation_steps can simulate larger batch sizes.
- Number of Epochs (num_train_epochs): How many times the model sees the entire training dataset.
  - Too Few: Underfitting (model doesn’t learn enough).
  - Too Many: Overfitting (model memorizes training data, performs poorly on unseen data). Monitor validation set performance to determine the optimal number of epochs.
- LoRA Parameters (lora_r, lora_alpha, lora_dropout): If using LoRA, these parameters control the behavior of the adapter layers.
  - lora_r (Rank): Higher rank = more trainable parameters, potentially better performance, but more memory. Experiment with values like 8, 16, 32.
  - lora_alpha: Scaling factor. Often set to 2*lora_r or other values.
  - lora_dropout: Regularization. Small values like 0.05 or 0.1 are common.
- Optimizer (optim): The algorithm used to update model weights.
  - AdamW (adamw_8bit): Generally a good default choice for LLMs.
  - Adafactor: Can be more memory-efficient for very large models.
- Learning Rate Scheduler (lr_scheduler_type): How the learning rate changes during training.
  - cosine: Common and effective for LLMs.
  - linear: Simple linear decay of learning rate.
  - constant_with_warmup: Warmup phase followed by a constant learning rate.
- Warmup Steps (warmup_steps): Number of steps to gradually increase the learning rate at the beginning of training.
Hyperparameter Tuning Strategies:
- Manual Search (Iterative Experimentation): Start with a set of reasonable hyperparameters, train, evaluate, analyze results, adjust hyperparameters based on observations, and repeat. This is often the most practical approach, especially when starting out.
- Grid Search: Define a grid of hyperparameter values and try all combinations. Computationally expensive for large hyperparameter spaces.
- Random Search: Randomly sample hyperparameter values from specified ranges. Often more efficient than grid search for high-dimensional hyperparameter spaces.
- Bayesian Optimization: Uses probabilistic models to intelligently guide the search for optimal hyperparameters, often requiring fewer trials than grid or random search. Libraries like Optuna and Ray Tune can automate Bayesian optimization with the Hugging Face Trainer.
- Hugging Face Trainer with hyperparameter_search: The Trainer class offers a built-in hyperparameter_search method that integrates with Optuna, Ray Tune, and other hyperparameter optimization libraries. This can automate the tuning process and make it more systematic.

Step 8: Advanced Fine-Tuning Techniques for 2025 – Pushing the Boundaries

For advanced users and more demanding tasks, explore these techniques:

Quantization – Squeezing Models for Efficiency: Quantization reduces the precision of model weights (e.g., from 32-bit floating-point to 8-bit integer or 4-bit integer). This dramatically reduces memory footprint and can speed up inference, allowing you to run larger models on less powerful hardware.
- Libraries: bitsandbytes, Unsloth (Unsloth offers optimized quantization and fine-tuning).
- Techniques: 8-bit quantization, 4-bit quantization (QLoRA).
Distributed Training – Scaling Up for Massive Models: For training very large models (70B+ parameters) or speeding up training with large datasets, distributed training is essential. It splits the training workload across multiple GPUs or machines.
- Libraries: Hugging Face accelerate, DeepSpeed, PyTorch Distributed Data Parallel (DDP).
- Techniques: Data parallelism, tensor parallelism, pipeline parallelism.
Mixed Precision Training (fp16, bf16) – Speed and Memory Savings: Uses a combination of 16-bit and 32-bit floating-point numbers during training. This speeds up computations (especially on GPUs with Tensor Cores or similar hardware) and reduces memory usage. The fp16=True and bf16=True arguments in TrainingArguments enable mixed precision.
Gradient Checkpointing – Memory Optimization for Long Sequences: Gradient checkpointing is a memory-saving technique that trades compute for memory. It recomputes intermediate activations during the backward pass instead of storing them, significantly reducing VRAM usage, especially for models with long context lengths. Enable it in your code:
```
model.gradient_checkpointing_enable()
```
Flash Attention – Faster and More Memory-Efficient Attention: Flash Attention and Flash Attention 2 are optimized attention mechanisms that significantly speed up training and reduce memory consumption, especially for long sequences.
- Library: flash-attn.
- Integration: Many models now support Flash Attention. Check your model’s documentation and ensure you have the correct flash-attn library version and hardware setup.
GRPO (Group Relative Policy Optimization) – Simplified and Fast RL Fine-tuning: As discussed earlier and demonstrated in the DataWizz AI article, GRPO is a promising alternative to traditional RLHF. It’s simpler, faster, and more stable, making RL-based fine-tuning more accessible, even for smaller models. Explore GRPO for tasks where you want to optimize for specific reward signals beyond supervised learning. The Hugging Face trl library provides a GRP Trainer.
RLHF (Reinforcement Learning from Human Feedback) & DPO – Advanced Alignment: For the highest level of alignment with human preferences and values, RLHF (or its simpler variant DPO) remains a powerful, though complex, approach. Use RLHF/DPO when you need fine-grained control over model behavior and want to optimize for subjective qualities like helpfulness, harmlessness, and truthfulness. The trl library provides tools for implementing RLHF and DPO pipelines.

Step 9: Saving and Loading Your Fine-Tuned Model – Persistence and Reusability

Saving and loading your model is essential for persisting your fine-tuning work and using the model for inference.

Saving Your Model (using Trainer):
```
trainer.save_model("my_fine_tuned_model")  # Saves model weights and config
tokenizer.save_pretrained("my_fine_tuned_model")  # Saves tokenizer files
```
This creates a directory (e.g., my_fine_tuned_model) containing:
- pytorch_model.bin: The fine-tuned model weights.
- config.json: Model configuration.
- tokenizer_config.json, tokenizer.model, special_tokens_map.json, vocab.json: Tokenizer files.

Loading Your Saved Model:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("my_fine_tuned_model", trust_remote_code=True) # Load fine-tuned model. trust_remote_code if needed by the model.
tokenizer = AutoTokenizer.from_pretrained("my_fine_tuned_model") # Load tokenizer.

Now you can use the loaded model and tokenizer for inference (text generation). Step 10: Complete End-to-End Fine-Tuning Workflow with Axolotl

Let’s walk through a complete workflow from fine-tuning to deployment using Axolotl.

1. Installation and Setup:

!git clone -q https://github.com/OpenAccess-AI-Collective/axolotl
%cd axolotl
pip install -e .

2. Create Configuration File (config.yaml): ```yaml # ====================================== # Model and Tokenizer Configuration # ====================================== base_model: Qwen/Qwen2.5-0.5B # Base model model_type: AutoModelForCausalLM # Specifies the model architecture. # Other options include AutoModelForSequenceClassification. tokenizer_type: AutoTokenizer # Tokenizer type. Use a custom tokenizer if needed. trust_remote_code: true # Enables loading custom code from remote repositories. # Required for models like Qwen that have custom implementations.

# ======================================
# Model Quantization and Loading (Optional)
# ======================================
# Uncomment these if you want to optimize memory usage.
# load_in_8bit: true                     # Enable 8-bit quantization; typically needs 16GB+ VRAM.
# load_in_4bit: false                    # 4-bit mode can be used in extreme memory constrained setups (<16GB).
# strict: false                          # When true, enforces strict config checks; false allows more flexibility.

# ======================================
# Dataset Configuration
# ======================================
datasets:
- path: HuggingFaceTB/smoltalk
    type: sharegpt                   # Data format; other formats: sharegpt:chat, json, csv, parquet.
    conversation: chatml             # ChatML conversation formatting; alternative: default/custom template.

# ======================================
# Dataset Preparation and Output
# ======================================
chat_template: chatml                   # Template for constructing conversational prompts.
                                        # Alternatives: custom prompts, or other DSLs.
dataset_prepared_path: qwen2.5-0.5b-data      # Directory where preprocessed dataset files are cached.
val_set_size: 0.01                       # Fraction (1%) of the dataset set aside for validation.
output_dir: qwen2.5-0.5b                      # Directory for storing fine-tuned model outputs, checkpoints, etc.

# ======================================
# Sequence and Batching Settings
# ======================================
sequence_len: 8192                        # Maximum token length supported by qwen2.5-0.5b.
                                        # Adjust lower if GPU memory is constrained.
sample_packing: true                      # Enables sample packing to maximize utilization.
pad_to_sequence_len: true                 # Pads sequences to the full length.
                                        # Alternatively, set to false to use dynamic batching.

# ======================================
# Fine-Tuning Strategy (Adapters / LoRA) [Optional]
# ======================================
# Uncomment these if you prefer adapter-based training, which fine-tunes only a subset of parameters.
# adapter: lora                          # Options include: lora, qlora, adalora for parameter-efficient tuning.
# lora_model_dir:                         # Path to existing LoRA weights if resuming.
# lora_r: 32                             # LoRA rank; alternative higher values may add capacity.
# lora_alpha: 16                         # Scaling factor; often set to approximately half of the base layers.
# lora_dropout: 0.05                     # Dropout rate; commonly between 0.0 and 0.1.
# lora_target_linear: true               # Whether to target linear layers for adaptation.
# lora_fan_in_fan_out:                   # Include if using fan-in/fan-out structured layers.

# ======================================
# Experiment Tracking (Weights & Biases)
# ======================================
wandb_project: qwen2.5-0.5b                 # Project name for experiment tracking.
wandb_entity:                           # Your wandb username or team; fill if using wandb.
wandb_watch:                            # Options: "gradients", "all" to log additional stats.
wandb_name:                             # (Optional) Run name for easier identification in wandb.
wandb_log_model:                        # Options: "checkpoint", "end", "all", or false to disable model logging.

# ======================================
# Training Hyperparameters
# ======================================
gradient_accumulation_steps: 8         # Increases the effective batch size by accumulating gradients.
micro_batch_size: 1                    # Batch size per GPU. If VRAM is limited, keeping this small is advised.
num_epochs: 3                          # Total number of epochs. Increase if you have more data or need further finetuning.
optimizer: paged_adamw_8bit            # Optimizer choice; alternatives include adamw_bnb_8bit, lion, adafactor.
lr_scheduler: cosine                   # Learning rate schedule; alternatives: linear, polynomial.
learning_rate: 1e-5                    # Starting learning rate; tune this based on dataset size and stability.
weight_decay: 0.05                     # Weight decay regularization; consider 0.0 if no regularization is needed.

# Data Preprocessing Options:
train_on_inputs: false                 # Whether to include the user prompt in the training data.
group_by_length: false                 # Group samples with similar lengths to reduce padding waste.
                                        # Set true if you want more efficient memory usage during batching.

# ======================================
# Precision and Performance Optimization
# ======================================
bf16: auto                             # Use bfloat16 if supported (auto) or explicitly set to true/false.
fp16: false                            # FP16 mixed precision (legacy); ensure only one of fp16 or bf16 is active.
tf32: false                            # Disables TensorFloat-32; set true for faster compute on compatible Ampere GPUs.
gradient_checkpointing: true           # Saves memory by re-computing activations; may slow training slightly.
xformers_attention:                     # Uncomment if using the xformers attention backend (requires installation).
flash_attention: true                  # Enable flash attention for supported GPUs (e.g., A100, H100, RTX 40xx).

# ======================================
# Scheduler, Logging, and Checkpointing Settings
# ======================================
warmup_steps: 10                       # Number of initial steps to gradually ramp up the learning rate.
evals_per_epoch: 2                     # Number of evaluations per epoch; increase for closer monitoring.
eval_table_size:                       # (Optional) Limit the number of samples in evaluation summary.
eval_max_new_tokens: 128               # Maximum tokens generated during evaluation; adjust according to task.
saves_per_epoch: 4                     # How many checkpoints are saved each epoch.
save_total_limit: 2                    # Maximum checkpoints to retain (older ones get deleted).
logging_steps: 1                       # Frequency of logging metrics (per step is recommended for debugging).

# ======================================
# Distributed and Advanced Training Options
# ======================================
deepspeed: /workspace/axolotl/deepspeed_configs/zero3_bf16_cpuoffload_params.json  
                                        # DeepSpeed configuration for advanced distributed training.
                                        # Alternatives include zero1, zero2, or custom DeepSpeed configs.
early_stopping_patience:               # (Optional) Number of evaluations with no improvement to trigger early stopping.
resume_from_checkpoint:                # (Optional) Path to resume training from a saved checkpoint.
local_rank:                            # Used in multi-GPU distributed training setups.
fsdp:                                  # Fully Sharded Data Parallel parameters, if using FSDP.
fsdp_config:                           # Additional configuration; see FSDP guidelines.

# ======================================
# Special Tokens (Customizable per Model)
# ======================================
special_tokens:
bos_token: "<|endoftext|>"           # Beginning-of-sequence token; alternatives: "<s>", "<bos>".
eos_token: "<|im_end|>"              # End-of-sequence token; alternatives: "

3. Run Fine-Tuning:

accelerate launch -m axolotl.cli.train config.yaml

4. Merge Adapter Weights (only if using LoRA/QLoRA):

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer
    
# Load the fine-tuned model
model = AutoPeftModelForCausalLM.from_pretrained(
    "output/fine_tuned_model",
    device_map="auto",
)
    
# Merge weights
merged_model = model.merge_and_unload()
    
# Save the merged model
merged_model.save_pretrained("merged_model")
tokenizer = AutoTokenizer.from_pretrained("output/fine_tuned_model")
tokenizer.save_pretrained("merged_model")

5. Push to Hugging Face Hub:

from huggingface_hub import HfApi
    
# Login to Hugging Face
api = HfApi()
api.login(token="your_token_here")
    
# Push model and tokenizer
merged_model.push_to_hub("your-username/model-name")
tokenizer.push_to_hub("your-username/model-name")

6. Create GGUF Format (for llama.cpp):

# Install llama-cpp-python
pip install llama-cpp-python
    
# Convert to GGUF
python -m llama_cpp.convert_hf_to_gguf \
    --outfile model_q4.gguf \
    --outtype q4_k_m \
    merged_model

7. Test the Model:

from transformers import AutoModelForCausalLM, AutoTokenizer
    
# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    "your-username/model-name",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("your-username/model-name")
    
# Test generation
prompt = "Your test prompt here"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Remember to adjust paths, model names, and configurations according to your specific needs.

Step 11: Deploying Your Fine-Tuned Model – Bringing Your AI to the World

Once you have a fine-tuned model, you need to deploy it to make it accessible for inference. Here are common deployment options:

Hugging Face Inference API: The simplest way to deploy your model is using Hugging Face’s Inference API. Upload your model to the Hugging Face Hub, and you can use their API to send requests and receive generated text. Good for quick demos and smaller-scale applications.
Local Inference (Self-Hosted): Run the model directly on your own servers or infrastructure. Provides full control and privacy. Use the transformers library and pipeline (as shown in Step 6’s evaluation example) to load and run your model locally. Libraries like vLLM and Text Generation Inference (TGI) can significantly optimize local inference performance.
Cloud-Based Deployment Platforms: Deploy your model on cloud platforms using specialized services: * Amazon SageMaker: AWS’s managed ML service, offering model deployment, scaling, and monitoring. * Google Cloud Vertex AI: Google Cloud’s ML platform, similar to SageMaker. * Microsoft Azure Machine Learning: Azure’s ML service, with deployment options. * vLLM (Fast Inference Library): vLLM is a highly optimized library specifically for LLM inference and serving. It offers significant speed and efficiency gains. Ideal for self-hosted deployments or cloud deployments where you need maximum performance. * Text Generation Inference (TGI) – Hugging Face’s Production Solution: TGI is Hugging Face’s purpose-built, production-ready solution for deploying and scaling LLMs. It’s highly optimized for performance, supports features like continuous batching, tensor parallelism, and integration with Hugging Face Hub. Recommended for production deployments, especially when scaling and efficiency are critical.

Step 12: Troubleshooting Common Fine-Tuning Issues – A 2025 Survival Guide

Fine-tuning can be challenging. Here are common problems and how to address them:

Out of Memory (OOM) Errors (GPU Memory Exhaustion):
- Reduce per_device_train_batch_size: Decrease the batch size.
- Increase gradient_accumulation_steps: Simulate larger batch sizes with less memory.
- Use a Smaller Model: Fine-tune a smaller base model.
- Enable PEFT (LoRA): Dramatically reduces memory usage compared to full fine-tuning.
- Enable Quantization (8-bit or 4-bit): Use libraries like bitsandbytes or Unsloth to quantize the model.
- Enable Gradient Checkpointing: Trade compute for memory.
- Reduce max_seq_length: Shorten the maximum sequence length.
- Use a GPU with More VRAM: If possible, upgrade to a GPU with more memory or use cloud GPUs.
- Sample Packing (Axolotl): If using Axolotl and sample packing, reducing sequence_len can help.
Slow Training:
- Increase per_device_train_batch_size (if memory allows): Larger batch sizes can improve throughput.
- Use a More Powerful GPU: Upgrade to a faster GPU.
- Enable Mixed Precision Training (fp16 or bf16): Can significantly speed up training on compatible GPUs.
- Use Distributed Training: Train across multiple GPUs or machines.
- Optimize Data Loading: Ensure data loading isn’t a bottleneck. Use num_proc in dataset.map to increase CPU preprocessing workers.
- Flash Attention 2: If supported, using Flash Attention 2 can drastically speed up training.
Poor Performance of Fine-Tuned Model:
- [CRITICAL] Check Your Prompt Template! Incorrect prompt formatting is the most frequent cause of poor performance. Double-check that you are using the correct prompt template for your chosen base model, and that you are using tokenizer.apply_chat_template() correctly.
- Data Quality Issues: Re-examine your dataset. Is it high-quality, relevant, and diverse? Address data quality problems (Step 3.4).
- Insufficient Data: You might need more training data, or consider data augmentation or synthetic data generation (Step 3.5).
- Hyperparameter Tuning: Experiment with different hyperparameters (learning rate, epochs, LoRA parameters, etc.) (Step 7).
- Base Model Choice: Try a different base model that might be better suited to your task (Step 2).
- LoRA target_modules (PEFT): If using LoRA, verify that your target_modules are correct for your chosen model architecture. Incorrect modules can severely degrade performance. Consider experimenting with lora_target_linear: true as an alternative.
- train_on_inputs: false (Axolotl/Chat Templates): If using chat templates and Axolotl, ensure train_on_inputs: false is set correctly to train only on assistant responses.
NaN Loss (Loss Becomes “Not a Number”):
- Reduce Learning Rate: Lower the learning_rate. A too-high learning rate can cause instability.
- Data Issues: Check your dataset for errors (e.g., division by zero, invalid numerical values).
- Optimizer Issues: Try a different optimizer or optimizer settings.
- Gradient Clipping: Enable gradient clipping in TrainingArguments (max_grad_norm=1.0).
- Mixed Precision Problems: If using bf16, try fp16 instead (or vice versa), or disable mixed precision temporarily for debugging.
Model Generates Nonsense or Repetitive Text:
- [CRITICAL] Verify Prompt Template: Again, double-check that you are using the correct prompt template!
- Dataset Format: Ensure your dataset is in the correct format (e.g., alternating user/assistant roles for conversational models).
- Insufficient Training: Train for more epochs (num_train_epochs).
- Generation Parameters: Experiment with generation parameters during inference (temperature, top_p, top_k, repetition penalty) to control the model’s output.
- Base Model Choice: Consider a different base model.

Key Takeaways & Best Practices for LLM Fine-Tuning in 2025:

Prompt Template is Paramount: Always, always, always use the correct prompt template for your chosen model. tokenizer.apply_chat_template() is your essential tool.
Data Quality is King (and Queen, and the Whole Court): Invest in creating a high-quality, relevant dataset. Small, high-quality datasets are better than large, noisy ones. Consider synthetic data generation for specialized tasks.
Start Small and Iterate: Begin with a smaller model and a smaller dataset for faster experimentation and iteration. Scale up model size and dataset size as needed.
Embrace PEFT (LoRA): Parameter-Efficient Fine-Tuning (especially LoRA) is highly recommended for efficiency, reduced resource requirements, and preventing catastrophic forgetting.
Monitor Training Closely: Use TensorBoard or Weights & Biases (WandB) to track training metrics (loss, evaluation metrics) and visualize progress. This is essential for debugging and hyperparameter tuning.
- Evaluate Thoroughly (Quantitatively and Qualitatively): Use both automated metrics and, crucially, human evaluation to assess your model’s performance. Don’t rely solely on loss curves. Inspect generated text manually.
- Hyperparameter Tuning is Not Optional: Experiment with different hyperparameters to find the optimal configuration for your task. Systematic tuning can significantly improve performance.
- train_on_inputs: false (Chat Templates): When using chat templates, ensure train_on_inputs: false in Axolotl or equivalent setting in other frameworks, so you’re training only on the assistant’s intended responses.
- Understand Quantization: Leverage quantization (8-bit, 4-bit) to reduce resource requirements and make larger models more accessible.
- Choose Hardware Wisely: Select GPUs with sufficient VRAM based on your model size and training parameters. Cloud GPUs offer flexibility and scalability.
- Stay Updated with the Latest Techniques: The field of LLM fine-tuning is rapidly evolving. Keep learning about new techniques, libraries, and best practices (e.g., GRPO, Flash Attention, Unsloth optimizations).

Further Exploration & Next Steps:

Dive Deeper into RLHF and GRPO: Experiment with Reinforcement Learning techniques using the trl library. Explore GRPO for simpler RL-based fine-tuning.
Explore Advanced PEFT Methods: Investigate other PEFT techniques beyond LoRA, such as Prefix Tuning, Prompt Tuning, and IA³.
Master Distributed Training: Learn how to scale your fine-tuning to multiple GPUs or machines using accelerate or DeepSpeed for training even larger models.
Contribute to the Community: Share your fine-tuned models, datasets, and code on the Hugging Face Hub to contribute to the open-source AI ecosystem. Engage with the community on forums and platforms like Discord to learn from others and share your experiences.
Stay Current with Research: Keep reading research papers and blog posts to stay informed about the latest advancements in LLMs and fine-tuning techniques. The field is moving incredibly fast!

This comprehensive guide provides a solid foundation for your journey into fine-tuning Large Language Models in 2025 and beyond. Start experimenting, iterate, and unlock the power of tailored AI for your specific applications!