--- type: "new-runtime-knowledge-guide" stable_id: "knowledge_guide:llm-finetuning" version: 1 generated_at: "2026-07-23" record_date: "2026-07-23" date_kind: "generated_at" slug: "llm-finetuning" title: "Fine-Tuning Strategies" description: "When to use fine-tuning, and when is RAG or prompting enough? LoRA, QLoRA, SFT, DPO, RLHF, distillation - broken down into focused cards. How to prepare data, how to evaluate the result, how not to waste the GPU." retrieval_nugget: "When to use fine-tuning, and when is RAG or prompting enough? LoRA, QLoRA, SFT, DPO, RLHF, distillation - broken down into focused cards. How to prepare data, how to evaluate the result, how not to waste the GPU. Fine-tuning solves different problems than RAG. They are often confused - and resources are wasted on the wrong things." track: "prompting-and-adaptation" volume: 4 --- # Fine-Tuning Strategies ## Retrieval answer When to use fine-tuning, and when is RAG or prompting enough? LoRA, QLoRA, SFT, DPO, RLHF, distillation - broken down into focused cards. How to prepare data, how to evaluate the result, how not to waste the GPU. Fine-tuning solves different problems than RAG. They are often confused - and resources are wasted on the wrong things. ## 01. The main question: should you fine-tune? ### 01.01 RAG vs Fine-tuning vs Prompting — decision tree Fine-tuning solves different problems than RAG. They are often confused - and resources are wasted on the wrong things. The key question is: what exactly is not working in the current system? #### Diagram ```text Problem: The model gives incorrect answers │ ├── The model does not know the current facts / company data │ └──→ RAG (add knowledge, leave weight alone) │ ├── The model does not understand the task the first time, long instructions are needed │ └──→ Prompt engineering + few-shot (try first) │ ├── The model does not produce the desired format/style/tone consistently │ └──→ SFT fine-tuning (train with examples of the correct format) │ ├── The model does not know how to do a specific task (domain reasoning) │ └──→ SFT + domain data (train on a task) │ ├── Model too expensive/slow for production │ └──→ Distillation (small model repeats large one) │ └── Model is toxic / ignores rules / does not follow policies └──→ DPO / RLHF (alignment) ``` ### 01.02 When RAG is better than fine tuning - 6 signs RAG is about knowledge. Fine-tuning is about behavior and skills. If the problem is knowledge, RAG is cheaper, faster and updated without retraining. #### RAG wins when ```text ✓ Data is updated frequently → re-indexing is cheaper than retraining ✓ You need a link to the source (citation) → RAG gives attribution out of the box ✓ Large corpus of specific facts → the model does not remember facts well ✓ No GPU or training data → RAG works with API models ✓ Different clients with different knowledge bases → one index per client, one model ✓ Need transparency/audit → sources are always visible ``` ### 01.03 When fine tuning is necessary - 6 signs Fine-tuning is justified when the problem is not in knowledge, but in the way the model reasons, responds or behaves - and this cannot be fixed by prompts. #### Fine-tuning is necessary when ```sql ✓ Need a specific style/tone consistently → “write as a lawyer for our company” ✓ The task requires domain reasoning → medical diagnostics, industrial calculations ✓ We need to remove the long few-shot from the prompt → saving tokens = reducing cost ✓ The problem cannot be solved by prompting at all → specific output format, rare language ✓ Need a small fast model (latency SLA) → GPT-4 distillation → 7B model ✓ Data is confidential and cannot be sent to the API → self-hosted fine-tuned model ``` ### 01.04 RAG + Fine-tuning - not OR, but AND The best production systems combine both approaches: fine tuning teaches the model to reason correctly and respond in the right style, RAG provides up-to-date knowledge. These are not competitors - these are different layers. #### Combined architecture ```sql fine-tuned model: ✓ knows the company's output format ✓ understands domain terminology ✓ knows how to reason in the right style ✗ does not know specific facts of clients + RAG on top: ✓ current data ✓ specific facts from the database ✓ links to sources → the result is better than each one separately ``` #### Combo examples - medical assistant: SFT on clinical reasoning + RAG on patient history - seller assistant: SFT on reporting format + RAG on WB/Ozon data ## 02. Supervised Fine-Tuning (SFT) ### 02.05 SFT - what's going on under the hood SFT (Supervised Fine-Tuning) - additional training of a pre-trained model on a dataset of pairs (input → output). The model can already do “everything” after pretraining; SFT does not add knowledge - it recalibrates the probability distribution to the desired answer format. This is supervised learning on top of an already trained model. #### Diagram ```sql Pretraining SFT Result ──────────────────────────── ──────────────────────────── Base model Fine-tuned model p(token | context) → updated to → stable trained on examples {input, output} required format the entire Internet from your dataset and style Loss: L = -Σ log p(output_token_i | input, output_tokens_# cross-entropy only on output tokens, not on input # input is masked → the model only learns to respond Chat template (important!): <|user|> {input} <|assistant|> {output} <|end|> # each model has its own format - use it exactly ``` #### When is SFT enough? - need a specific output format - domain-specific task with examples - remove long few-shot from prompt - need 500–5000 high-quality examples ### 02.06 Instruction Tuning - SFT for following instructions Special form of SFT: learning from a large set of varied instructions. This is what turns the base model into a chat model. FLAN, Alpaca, ShareGPT - datasets for instruction tuning. #### instruction tuning data format ```text { "instruction": "Translate to English", "input": "Hi, how are you?", "output": "Hello, how are you?" } # thousands of such examples for different tasks # → the model learns to follow any instruction Key: variety of tasks is more important than volume 1000 different instructions > 10000 of the same type ``` ### 02.07 Catastrophic Forgetting is the main danger of SFT When fine tuning, the model may “forget” the general abilities that were in the base model. It is especially dangerous with a small dataset and a large learning rate. The model does the new task well, but everything else is bad. #### How to prevent ```text ✓ Low learning rate 1e-5 – 3e-5 for full fine-tuning # don't break what already worked ✓ Small dataset → LoRA instead of full FT # update minimum parameters ✓ Replay: mix with general-purpose data # 10–20% of the original data in the dataset ✓ Early stopping by validation loss # do not retrain to zero ✗ Many epochs on a small dataset # → retraining, loss of general abilities ``` ## 03. LoRA & Parameter-Efficient Fine-Tuning ### 03.08 LoRA - Low-Rank Adaptation, atomic LoRA is the most popular method of effective fine tuning. Idea: do not update all the model weights (billions of parameters), but add small low-rank matrices on top of the key layers. Only they are updated - 0.1–1% of the parameters from the original model. #### Diagram ```sql Regular layer: W (d×d matrix, frozen) LoRA adds: ΔW = A × B where A: d×r, B: r×d, r ≪ d (rank) Layer output: h = W x + ΔW x = W x + (A B) x Example for 7B model: Full FT: 7,000,000,000 parameters are updated LoRA r=16: 4,000,000 parameters updated → 0.06% After training - merge back to W: W_new = W + A B ← no overhead in inference or store A, B separately → swap adapters for different tasks ``` #### Parameter rank (r) - how to choose - r=4–8: simple tasks, format - r=16–32: standard, most tasks - r=64–128: complex domain reasoning - r higher → more parameters, risk of overfitting ### 03.09 QLoRA - fine tuning on a consumer GPU QLoRA = LoRA + base model quantization in 4-bit (NF4). The base model is loaded in 4-bit (frozen), LoRA adapters are trained in bf16. Allows fine-tuning 13B on 1× RTX 3090, 70B on 2× A100. #### Memory Requirements ```text Model Full FT LoRA QLoRA ───────────────────── ───────────────────── 7B parameters 28 GB 14 GB 6 GB ← RTX 3090 13B parameters 52 GB 26 GB 10 GB ← RTX 4090 34B parameters 136 GB 68 GB 22 GB ← A100 40G 70B parameters 280 GB 140 GB 48 GB ← 2×A100 QLoRA quality loss vs LoRA: ~1-2% on most tasks ``` ### 03.10 Other PEFT methods - DoRA, IA³, Prefix Tuning LoRA is not the only PEFT method. Other approaches may be suitable for different problems and constraints. All implemented in the PEFT library from HuggingFace. #### Alternatives to LoRA ```text DoRA (Weight-Decomposed LoRA) separates magnitude and direction updates → slightly better than LoRA with the same rank IA³ (Infused Adapter by Inhibiting and Amplifying) scales activations, does not add matrices → even fewer parameters than LoRA Prefix Tuning / Prompt Tuning adds trainees "soft tokens" to the input → does not change the weight at all, few parameters → performs worse than LoRA on complex tasks In practice: LoRA/QLoRA → default choice ``` ### 03.11 LoRA Adapters - swap without retraining LoRA adapters can be stored separately from the base model and loaded dynamically. One basic model + N adapters for N tasks - saving memory, fast switching. These are like plugins for the model. #### Multi-adapter serving ```text base_model = load("Llama-3.1-8B") # 1 time, 8GB adapter_legal = load_lora("legal_v2") # 50MB adapter_medical = load_lora("medical_v1") # 50MB adapter_seller = load_lora("wb_seller") # 50MB # at runtime on request: model.set_adapter(route_to_adapter(query)) output = model.generate(query) # LoRAX, vLLM, S-LoRA - servers with support ``` ## 04. Alignment — RLHF, DPO, ORPO ### 04.12 RLHF — Reinforcement Learning from Human Feedback RLHF is a three-step process that GPT-4, Claude, Gemini are trained on. This is not just “learning from feedback” - it is a complex pipeline with three separate models. It is RLHF that makes models useful and safe. #### Diagram ```sql Stage 1: SFT (Supervised Fine-Tuning) base_model + demos from people → SFT model Stage 2: Reward Model Training people rank pairs of answers: A > B trains Reward Model to predict human preferences reward(prompt, response) → scalar score Stage 3: RL with PPO (Proximal Policy Optimization) SFT model = policy (what we train) Reward Model = environment (gives a reward) + KL-penalty vs original SFT model (don’t go far) L = reward(response) - β·KL(policy || SFT_ref) RLHF problem: difficult, unstable, requires many GPUs → DPO solves this more elegantly ``` ### 04.13 DPO - Direct Preference Optimization, atomic DPO (2023) is a mathematically equivalent solution to RLHF without a separate Reward Model and without RL. Key idea: it turns out that the RL problem can be reformulated as a supervised learning problem directly from preference data. It's elegant mathematics that simplifies everything. #### Diagram ```text Data: triples (prompt, chosen_response, rejected_response) RLHF: prompt → [RM, PPO, KL, instability...] → aligned model DPO: prompt + chosen + rejected → one gradient step → aligned model DPO Loss (simplified): L = -log σ(β log[π(chosen)/π_ref(chosen)] - β log[π(rejected)/π_ref(rejected)]) Meaning: increase the probability of chosen, reduce the probability of rejected relative to reference model (original SFT) Why DPO wins in practice: ✓ No Reward Model (one model instead of three) ✓ Stable training (supervised loss) ✓ Fewer GPUs, easier to implement ✓ Comparable quality to RLHF ``` #### Data for DPO - UltraFeedback, Anthropic HH-RLHF - or generate: GPT-4 → chosen, bad model → rejected ### 04.14 ORPO - one pass instead of two ORPO (Odds Ratio Preference Optimization, 2024) combines SFT and DPO in one step. There is no need to do SFT first, then DPO - learning alignment occurs simultaneously with learning to follow instructions. #### Pipeline comparison ```sql RLHF: pretraining → SFT → RM → PPO # 4 stages DPO: pretraining → SFT → DPO # 3 stages ORPO: pretraining → ORPO # 2 stages ORPO Loss: L = L_SFT + λ L_OR where L_OR = log odds ratio(chosen vs rejected) Practice: DPO - standard, ORPO - if there is no SFT checkpoint ``` ### 04.15 Constitutional AI / RLAIF Instead of human labelers, the LLM judge generates preference data. RLAIF (RL from AI Feedback, Anthropic) - Reward Model is trained on the estimates of a strong model. More scalable and cheaper than human markup. #### Pipeline RLAIF ```python # Instead of people: for prompt in dataset: response_A = model.generate(prompt) response_B = model.generate(prompt) preference = judge_llm(prompt, A, B) # "A is better because..." pairs.append((prompt, A, B, preference)) # Next is the standard DPO on these pairs train_dpo(model, pairs) ``` #### Applicable - no budget for markers - you need a large dataset of preferences ## 05. Distillation - small model as big ### 05.16 Knowledge Distillation - atomic Distillation is training a small model (student) to imitate a large one (teacher). Goal: get 80–90% teacher quality with 10x smaller inference size and cost. The basis for production optimization of LLM pipelines. #### Diagram ```text Teacher: GPT-4 / Claude Opus ← expensive, slow, API-only Student: Llama-3.1-8B ← cheap, fast, self-hosted Classic distillation (Hinton 2015): Loss = α CE(student_logits, labels) # hard labels + (1-α) · KL(student || teacher) · T² # soft labels # T = temperature: the higher, the “softer” the teacher distribution # soft labels carry more information than one-hot For LLM tasks - data distillation (simpler, works well): 1. Run teacher on N prompts → generate answers 2. SFT student on these pairs (prompt → teacher_answer) # teacher student learns to be like without access to logits ``` #### Applicable - reduce cost by 10–50x - latency < 100ms requires a small model - self-hosted without API dependency ### 05.17 Specialize & Distill - practical pattern Not trying to distill “everything” from teacher is ineffective. Distill only a specific task: teacher generates synthetic data for a task, student learns only on it. A small specialized model beats a large general one. #### Practical pipeline ```python # 1. Collect or generate task prompts prompts = domain_prompts + synthetic_generated(N=5000) #2. Teacher generates answers (expensive, one time) dataset = [(p, gpt4(p)) for p in prompts] #3. SFT of a small model on this data student = finetune("Llama-3.1-8B", dataset, epochs=3) #4. Eval: compare quality vs cost # Result: quality ~85% teacher, cost 1/30 ``` ### 05.18 Reasoning Distillation - chain-of-thought distillation Distill not only the answer, but also the chain of reasoning teacher. Student learns to reason like a large model. This is the basis of models like DeepSeek-R1-Distill - small models with strong reasoning. #### Scheme ```text teacher: o1 / DeepSeek-R1 / QwQ → generates: ...long argument... response dataset: (prompt, reasoning + answer) student: Llama/Qwen 7–14B → SFT on full traces with reasoning result: DeepSeek-R1-Distill-Qwen-7B ≈ 70% quality o1 at 1/100 cost ``` ## 06. Data is the bottleneck of all fine tuning ### 06.19 How much data is needed - practical numbers The main myth: “fine-tuning requires millions of examples.” The reality is different - a small, high-quality dataset is often better than a large, bad one. Quality >> quantity. #### Diagram ```text Objective Minimum Good Excellent ──────────────────────────────── ──────────────────────────────── Output style/format 100–200 500 1000+ Classification (4–10 classes) 200–500 1000 5000+ Entity extraction 500 2000 10000+ Specific domain QA 500 2000 5000+ Instructions following 1000 5000 50000+ DPO alignment 500 pairs 2000 pairs 10000+ pairs Distillation of a narrow task 1000 5000 20000+ The main rule: 500 high-quality > 10,000 noisy ``` ### 06.20 Synthetic Data Generation - dataset scaling No data - LLM will generate synthetic ones. Self-Instruct, Evol-Instruct, Magpie - patterns for generating training data through strong models. Manual sample verification is required. #### Self-Instruct pipeline ```python # 1. Seed: 20-30 real quality examples seed_examples = human_curated_examples #2. LLM generates new instructions new_instructions = GPT4(f""" Here are examples of tasks: {seed_examples} Generate 20 new similar but different problems. """) #3. LLM responds to generated instructions dataset = [(instr, GPT4(instr)) for instr in new_instructions] #4. Filtering: deduplication + quality filter filtered = [x for x in dataset if quality_score(x) > 0.7] ``` ### 06.21 Data Quality - what spoils a dataset Bad data is the main reason why fine tuning fails. The model will learn exactly what is in the data—including errors, inconsistencies, and noise. Garbage in - garbage out literally works here. #### Common data problems ```text Inconsistency There are different correct answers to the same question → the model cannot learn the pattern Duplication one example 100 times → the model retrains it → deduplicate by embedding similarity Leaked response prompt output contains instruction parts → mask instruction correctly Invalid chat template <|user|> and <|assistant|> tokens are mixed up → the model learns the wrong roles ✓ Verification: manually review 100 random examples ``` ### 06.22 Data Curation Pipeline - from raw materials to dataset Data preparation takes 60–80% of fine tuning time. We need a reproducible pipeline with versioning, otherwise it is impossible to understand what produced the result. #### Pipeline ```text raw_data → collect logs, databases, synthetics, expert examples → clean remove noise, special characters, artifacts → deduplicate MinHash / embedding similarity → filter LLM-quality-score > threshold → model-specific format chat template → split train 90% / val 5% / test 5% → version git + DVC / HuggingFace Datasets → audit manual check of 100 examples ``` ## 07. Infrastructure and tools ### 07.23 Fine-tuning tools - map The ecosystem of tools is divided into levels: low-level (HF Transformers), high-level (Axolotl, LLaMA-Factory), managed (Modal, Together, OpenAI FT API). The choice comes down to control vs convenience. #### Diagram ```text Level Tool Control Convenience Cost ─────────────────────────────────── ─────────────────────────────────── Low-level HF Transformers maximum minimum GPU in fact PEFT library TRL (SFT/DPO) High-level Axolotl high high GPU in fact LLaMA-Factory ← I recommend for starting Unsloth ← 2x faster, less VRAM Managed Together.ai FT low maximum $$/hour OpenAI FT API ← GPT-3.5/4o-mini only Modal + Axolotl medium high serverless GPU Eval during FT W&B / Langfuse log loss, eval metrics ``` ### 07.24 Hyperparameters - what really matters Most hyperparameters are not critical - there are 5 parameters that determine 90% of the result. The rest is fine tuning after the basic launch works. #### Starting values ```text learning_rate: 2e-4 #LoRA; 1e-5 for full FT lr_scheduler: cosine with warmup warmup_ratio: 0.05 # 5% steps to warm up epochs: 1–3 # rarely need more than 3 batch_size: per_device=2, grad_accum=8 # effective=16 max_seq_length: 2048 # per task LoRA specific: r: 16# rank alpha: 32 # = 2×r usually target_modules: ["q_proj","v_proj","k_proj","o_proj"] dropout: 0.05 ``` ### 07.25 Eval during and after fine tuning Fine-tuning without eval is learning blindly. You need to monitor: train/val loss (overfitting?), target metric for holdout (real quality), general abilities (have they degraded?). #### Checklist ratings ```text During training: ✓ is train_loss decreasing? ✓ val_loss is not growing? (if growing → overfitting) ✓ Is grad_norm stable? (spike → lr too high) After training: ✓ target_metric on test set vs baseline ✓ general benchmarks have not fallen? (MT-Bench, MMLU, HumanEval) ✓ human evaluation 50 examples ✓ compare vs prompted GPT-4 on the same task ``` ## 08. Final strategy map ### 08.26 Comparison of all approaches Summary table for quick navigation through strategies. | Method | Solves | Data | GPU | Complexity | | --- | --- | --- | --- | --- | | Prompting | Format and instructions | 0 | No | Low | | RAG | Knowledge and facts | Index | No | Medium | | SFT (LoRA) | Style and task behavior | 500–5k | 1×A100 | Medium | | SFT (full) | Deep domain adaptation | 10k+ | 8×A100 | High | | DPO | Behavior and safety | 500+ pairs | 1×A100 | Medium | | Distillation | Cost and latency | 1k–20k | 1×A100 | Medium | | RAG + SFT | Knowledge and style | Both | 1×A100 | High | ### 08.27 Rule of three questions Before any fine tuning project, answer three questions. If there’s even one “no,” go back a step. #### Checklist before start ```text 1. Have you exhausted your prompting? Have you tried few-shot, CoT, persona, chain? No → prompting first, it's free 2. Do you have data and metrics? Minimum 200 quality examples? A clear metric by which to measure improvement? No → first collect data and define a metric 3. Is the problem in behavior, not knowledge? If you add the correct data to the prompt → does everything work? Yes → use RAG, not fine tuning All three yes? Fine-tune. Start with LoRA on 7-8B models. Compare with prompted GPT-4 on your task. ```