Fine-Tuning Strategies

Problem: The model gives incorrect answers
 │
 ├── The model does not know the current facts / company data
 │ └──→ RAG (add knowledge, leave weight alone)
 │
 ├── The model does not understand the task the first time, long instructions are needed
 │ └──→ Prompt engineering + few-shot (try first)
 │
 ├── The model does not produce the desired format/style/tone consistently
 │ └──→ SFT fine-tuning (train with examples of the correct format)
 │
 ├── The model does not know how to do a specific task (domain reasoning)
 │ └──→ SFT + domain data (train on a task)
 │
 ├── Model too expensive/slow for production
 │ └──→ Distillation (small model repeats large one)
 │
 └── Model is toxic / ignores rules / does not follow policies
         └──→ DPO / RLHF (alignment)

✓ Data is updated frequently
  → re-indexing is cheaper than retraining

✓ You need a link to the source (citation)
  → RAG gives attribution out of the box

✓ Large corpus of specific facts
  → the model does not remember facts well

✓ No GPU or training data
  → RAG works with API models

✓ Different clients with different knowledge bases
  → one index per client, one model

✓ Need transparency/audit
  → sources are always visible

✓ Need a specific style/tone consistently
  → “write as a lawyer for our company”

✓ The task requires domain reasoning
  → medical diagnostics, industrial calculations

✓ We need to remove the long few-shot from the prompt
  → saving tokens = reducing cost

✓ The problem cannot be solved by prompting at all
  → specific output format, rare language

✓ Need a small fast model (latency SLA)
  → GPT-4 distillation → 7B model

✓ Data is confidential and cannot be sent to the API
  → self-hosted fine-tuned model

fine-tuned model:
  ✓ knows the company's output format
  ✓ understands domain terminology
  ✓ knows how to reason in the right style
  ✗ does not know specific facts of clients

+ RAG on top:
  ✓ current data
  ✓ specific facts from the database
  ✓ links to sources

→ the result is better than each one separately

Pretraining SFT Result
──────────────────────────── ────────────────────────────
Base model Fine-tuned model
  p(token | context) → updated to → stable
  trained on examples {input, output} required format
  the entire Internet from your dataset and style

Loss:
  L = -Σ log p(output_token_i | input, output_tokens_# cross-entropy only on output tokens, not on input
  # input is masked → the model only learns to respond

Chat template (important!):
  <|user|> {input} <|assistant|> {output} <|end|>
  # each model has its own format - use it exactly

{
  "instruction": "Translate to English",
  "input": "Hi, how are you?",
  "output": "Hello, how are you?"
}
# thousands of such examples for different tasks
# → the model learns to follow any instruction

Key: variety of tasks is more important than volume
  1000 different instructions > 10000 of the same type

✓ Low learning rate
  1e-5 – 3e-5 for full fine-tuning
  # don't break what already worked

✓ Small dataset → LoRA instead of full FT
  # update minimum parameters

✓ Replay: mix with general-purpose data
  # 10–20% of the original data in the dataset

✓ Early stopping by validation loss
  # do not retrain to zero

✗ Many epochs on a small dataset
  # → retraining, loss of general abilities

Regular layer: W (d×d matrix, frozen)
LoRA adds: ΔW = A × B
  where A: d×r, B: r×d, r ≪ d (rank)

Layer output: h = W x + ΔW x = W x + (A B) x

Example for 7B model:
  Full FT: 7,000,000,000 parameters are updated
  LoRA r=16: 4,000,000 parameters updated → 0.06%

After training - merge back to W:
W_new = W + A B ← no overhead in inference
or store A, B separately → swap adapters for different tasks

Model Full FT LoRA QLoRA
───────────────────── ─────────────────────
7B parameters 28 GB 14 GB 6 GB ← RTX 3090
13B parameters 52 GB 26 GB 10 GB ← RTX 4090
34B parameters 136 GB 68 GB 22 GB ← A100 40G
70B parameters 280 GB 140 GB 48 GB ← 2×A100

QLoRA quality loss vs LoRA: ~1-2% on most tasks

DoRA (Weight-Decomposed LoRA)
  separates magnitude and direction updates
  → slightly better than LoRA with the same rank

IA³ (Infused Adapter by Inhibiting and Amplifying)
  scales activations, does not add matrices
  → even fewer parameters than LoRA

Prefix Tuning / Prompt Tuning
  adds trainees "soft tokens" to the input
  → does not change the weight at all, few parameters
  → performs worse than LoRA on complex tasks

In practice: LoRA/QLoRA → default choice

base_model = load("Llama-3.1-8B") # 1 time, 8GB

adapter_legal = load_lora("legal_v2") # 50MB
adapter_medical = load_lora("medical_v1") # 50MB
adapter_seller = load_lora("wb_seller") # 50MB

# at runtime on request:
model.set_adapter(route_to_adapter(query))
output = model.generate(query)

# LoRAX, vLLM, S-LoRA - servers with support

Stage 1: SFT (Supervised Fine-Tuning)
  base_model + demos from people → SFT model

Stage 2: Reward Model Training
  people rank pairs of answers: A > B
  trains Reward Model to predict human preferences
  reward(prompt, response) → scalar score

Stage 3: RL with PPO (Proximal Policy Optimization)
  SFT model = policy (what we train)
  Reward Model = environment (gives a reward)
  + KL-penalty vs original SFT model (don’t go far)
  L = reward(response) - β·KL(policy || SFT_ref)

RLHF problem: difficult, unstable, requires many GPUs
→ DPO solves this more elegantly

Data: triples (prompt, chosen_response, rejected_response)

RLHF: prompt → [RM, PPO, KL, instability...] → aligned model
DPO: prompt + chosen + rejected → one gradient step → aligned model

DPO Loss (simplified):
L = -log σ(β log[π(chosen)/π_ref(chosen)]
         - β log[π(rejected)/π_ref(rejected)])

Meaning: increase the probability of chosen, reduce the probability of rejected
       relative to reference model (original SFT)

Why DPO wins in practice:
  ✓ No Reward Model (one model instead of three)
  ✓ Stable training (supervised loss)
  ✓ Fewer GPUs, easier to implement
  ✓ Comparable quality to RLHF

RLHF: pretraining → SFT → RM → PPO # 4 stages
DPO: pretraining → SFT → DPO # 3 stages
ORPO: pretraining → ORPO # 2 stages

ORPO Loss:
L = L_SFT + λ L_OR
where L_OR = log odds ratio(chosen vs rejected)

Practice: DPO - standard, ORPO - if there is no SFT checkpoint

# Instead of people:
for prompt in dataset:
    response_A = model.generate(prompt)
    response_B = model.generate(prompt)
    preference = judge_llm(prompt, A, B)
    # "A is better because..."
    pairs.append((prompt, A, B, preference))

# Next is the standard DPO on these pairs
train_dpo(model, pairs)

Teacher: GPT-4 / Claude Opus ← expensive, slow, API-only
Student: Llama-3.1-8B ← cheap, fast, self-hosted

Classic distillation (Hinton 2015):
  Loss = α CE(student_logits, labels) # hard labels
       + (1-α) · KL(student || teacher) · T² # soft labels
  # T = temperature: the higher, the “softer” the teacher distribution
  # soft labels carry more information than one-hot

For LLM tasks - data distillation (simpler, works well):
1. Run teacher on N prompts → generate answers
2. SFT student on these pairs (prompt → teacher_answer)
   # teacher student learns to be like without access to logits

# 1. Collect or generate task prompts
prompts = domain_prompts + synthetic_generated(N=5000)

#2. Teacher generates answers (expensive, one time)
dataset = [(p, gpt4(p)) for p in prompts]

#3. SFT of a small model on this data
student = finetune("Llama-3.1-8B", dataset, epochs=3)

#4. Eval: compare quality vs cost
# Result: quality ~85% teacher, cost 1/30

teacher: o1 / DeepSeek-R1 / QwQ
→ generates: <think>...long argument...</think> response

dataset: (prompt, <think>reasoning</think> + answer)

student: Llama/Qwen 7–14B
→ SFT on full traces with reasoning

result: DeepSeek-R1-Distill-Qwen-7B
  ≈ 70% quality o1 at 1/100 cost

Objective Minimum Good Excellent
──────────────────────────────── ────────────────────────────────
Output style/format 100–200 500 1000+
Classification (4–10 classes) 200–500 1000 5000+
Entity extraction 500 2000 10000+
Specific domain QA 500 2000 5000+
Instructions following 1000 5000 50000+
DPO alignment 500 pairs 2000 pairs 10000+ pairs
Distillation of a narrow task 1000 5000 20000+

The main rule: 500 high-quality > 10,000 noisy

# 1. Seed: 20-30 real quality examples
seed_examples = human_curated_examples

#2. LLM generates new instructions
new_instructions = GPT4(f"""
    Here are examples of tasks: {seed_examples}
    Generate 20 new similar but different problems.
""")

#3. LLM responds to generated instructions
dataset = [(instr, GPT4(instr)) for instr in new_instructions]

#4. Filtering: deduplication + quality filter
filtered = [x for x in dataset if quality_score(x) > 0.7]

Inconsistency
  There are different correct answers to the same question
  → the model cannot learn the pattern

Duplication
  one example 100 times → the model retrains it
  → deduplicate by embedding similarity

Leaked response prompt
  output contains instruction parts
  → mask instruction correctly

Invalid chat template
  <|user|> and <|assistant|> tokens are mixed up
  → the model learns the wrong roles

✓ Verification: manually review 100 random examples

raw_data
 → collect logs, databases, synthetics, expert examples
 → clean remove noise, special characters, artifacts
 → deduplicate MinHash / embedding similarity
 → filter LLM-quality-score > threshold
 → model-specific format chat template
 → split train 90% / val 5% / test 5%
 → version git + DVC / HuggingFace Datasets
 → audit manual check of 100 examples

Level Tool Control Convenience Cost
─────────────────────────────────── ───────────────────────────────────
Low-level HF Transformers maximum minimum GPU in fact
                   PEFT library
                   TRL (SFT/DPO)

High-level Axolotl high high GPU in fact
                   LLaMA-Factory ← I recommend for starting
                   Unsloth ← 2x faster, less VRAM

Managed Together.ai FT low maximum $$/hour
                   OpenAI FT API ← GPT-3.5/4o-mini only
                   Modal + Axolotl medium high serverless GPU

Eval during FT W&B / Langfuse log loss, eval metrics

learning_rate: 2e-4 #LoRA; 1e-5 for full FT
lr_scheduler: cosine with warmup
warmup_ratio: 0.05 # 5% steps to warm up
epochs: 1–3 # rarely need more than 3
batch_size: per_device=2, grad_accum=8 # effective=16
max_seq_length: 2048 # per task

LoRA specific:
  r: 16# rank
  alpha: 32 # = 2×r usually
  target_modules: ["q_proj","v_proj","k_proj","o_proj"]
  dropout: 0.05

During training:
  ✓ is train_loss decreasing?
  ✓ val_loss is not growing? (if growing → overfitting)
  ✓ Is grad_norm stable? (spike → lr too high)

After training:
  ✓ target_metric on test set vs baseline
  ✓ general benchmarks have not fallen?
     (MT-Bench, MMLU, HumanEval)
  ✓ human evaluation 50 examples
  ✓ compare vs prompted GPT-4 on the same task

1. Have you exhausted your prompting?
   Have you tried few-shot, CoT, persona, chain?
   No → prompting first, it's free

2. Do you have data and metrics?
   Minimum 200 quality examples?
   A clear metric by which to measure improvement?
   No → first collect data and define a metric

3. Is the problem in behavior, not knowledge?
   If you add the correct data to the prompt → does everything work?
   Yes → use RAG, not fine tuning

All three yes? FineTun.
Start with LoRA on 7-8B models.
Compare with prompted GPT-4 on your task.