Fine-Tuning Strategies

✓ Data is updated frequently
  → re-indexing is cheaper than retraining

✓ You need a link to the source (citation)
  → RAG gives attribution out of the box

✓ Large corpus of specific facts
  → the model does not remember facts well

✓ No GPU or training data
  → RAG works with API models

✓ Different clients with different knowledge bases
  → one index per client, one model

✓ Need transparency/audit
  → sources are always visible

✓ Need a specific style/tone consistently
  → “write as a lawyer for our company”

✓ The task requires domain reasoning
  → medical diagnostics, industrial calculations

✓ We need to remove the long few-shot from the prompt
  → saving tokens = reducing cost

✓ The problem cannot be solved by prompting at all
  → specific output format, rare language

✓ Need a small fast model (latency SLA)
  → GPT-4 distillation → 7B model

✓ Data is confidential and cannot be sent to the API
  → self-hosted fine-tuned model

fine-tuned model:
  ✓ knows the company's output format
  ✓ understands domain terminology
  ✓ knows how to reason in the right style
  ✗ does not know specific facts of clients

+ RAG on top:
  ✓ current data
  ✓ specific facts from the database
  ✓ links to sources

→ the result is better than each one separately

{
  "instruction": "Translate to English",
  "input": "Hi, how are you?",
  "output": "Hello, how are you?"
}
# thousands of such examples for different tasks
# → the model learns to follow any instruction

Key: variety of tasks is more important than volume
  1000 different instructions > 10000 of the same type

✓ Low learning rate
  1e-5 – 3e-5 for full fine-tuning
  # don't break what already worked

✓ Small dataset → LoRA instead of full FT
  # update minimum parameters

✓ Replay: mix with general-purpose data
  # 10–20% of the original data in the dataset

✓ Early stopping by validation loss
  # do not retrain to zero

✗ Many epochs on a small dataset
  # → retraining, loss of general abilities

Model Full FT LoRA QLoRA
───────────────────── ─────────────────────
7B parameters 28 GB 14 GB 6 GB ← RTX 3090
13B parameters 52 GB 26 GB 10 GB ← RTX 4090
34B parameters 136 GB 68 GB 22 GB ← A100 40G
70B parameters 280 GB 140 GB 48 GB ← 2×A100

QLoRA quality loss vs LoRA: ~1-2% on most tasks

DoRA (Weight-Decomposed LoRA)
  separates magnitude and direction updates
  → slightly better than LoRA with the same rank

IA³ (Infused Adapter by Inhibiting and Amplifying)
  scales activations, does not add matrices
  → even fewer parameters than LoRA

Prefix Tuning / Prompt Tuning
  adds trainees "soft tokens" to the input
  → does not change the weight at all, few parameters
  → performs worse than LoRA on complex tasks

In practice: LoRA/QLoRA → default choice

base_model = load("Llama-3.1-8B") # 1 time, 8GB

adapter_legal = load_lora("legal_v2") # 50MB
adapter_medical = load_lora("medical_v1") # 50MB
adapter_seller = load_lora("wb_seller") # 50MB

# at runtime on request:
model.set_adapter(route_to_adapter(query))
output = model.generate(query)

# LoRAX, vLLM, S-LoRA - servers with support

RLHF: pretraining → SFT → RM → PPO # 4 stages
DPO: pretraining → SFT → DPO # 3 stages
ORPO: pretraining → ORPO # 2 stages

ORPO Loss:
L = L_SFT + λ L_OR
where L_OR = log odds ratio(chosen vs rejected)

Practice: DPO - standard, ORPO - if there is no SFT checkpoint

# Instead of people:
for prompt in dataset:
    response_A = model.generate(prompt)
    response_B = model.generate(prompt)
    preference = judge_llm(prompt, A, B)
    # "A is better because..."
    pairs.append((prompt, A, B, preference))

# Next is the standard DPO on these pairs
train_dpo(model, pairs)

# 1. Collect or generate task prompts
prompts = domain_prompts + synthetic_generated(N=5000)

#2. Teacher generates answers (expensive, one time)
dataset = [(p, gpt4(p)) for p in prompts]

#3. SFT of a small model on this data
student = finetune("Llama-3.1-8B", dataset, epochs=3)

#4. Eval: compare quality vs cost
# Result: quality ~85% teacher, cost 1/30

teacher: o1 / DeepSeek-R1 / QwQ
→ generates: <think>...long argument...</think> response

dataset: (prompt, <think>reasoning</think> + answer)

student: Llama/Qwen 7–14B
→ SFT on full traces with reasoning

result: DeepSeek-R1-Distill-Qwen-7B
  ≈ 70% quality o1 at 1/100 cost

# 1. Seed: 20-30 real quality examples
seed_examples = human_curated_examples

#2. LLM generates new instructions
new_instructions = GPT4(f"""
    Here are examples of tasks: {seed_examples}
    Generate 20 new similar but different problems.
""")

#3. LLM responds to generated instructions
dataset = [(instr, GPT4(instr)) for instr in new_instructions]

#4. Filtering: deduplication + quality filter
filtered = [x for x in dataset if quality_score(x) > 0.7]

Inconsistency
  There are different correct answers to the same question
  → the model cannot learn the pattern

Duplication
  one example 100 times → the model retrains it
  → deduplicate by embedding similarity

Leaked response prompt
  output contains instruction parts
  → mask instruction correctly

Invalid chat template
  <|user|> and <|assistant|> tokens are mixed up
  → the model learns the wrong roles

✓ Verification: manually review 100 random examples

raw_data
 → collect logs, databases, synthetics, expert examples
 → clean remove noise, special characters, artifacts
 → deduplicate MinHash / embedding similarity
 → filter LLM-quality-score > threshold
 → model-specific format chat template
 → split train 90% / val 5% / test 5%
 → version git + DVC / HuggingFace Datasets
 → audit manual check of 100 examples

learning_rate: 2e-4 #LoRA; 1e-5 for full FT
lr_scheduler: cosine with warmup
warmup_ratio: 0.05 # 5% steps to warm up
epochs: 1–3 # rarely need more than 3
batch_size: per_device=2, grad_accum=8 # effective=16
max_seq_length: 2048 # per task

LoRA specific:
  r: 16# rank
  alpha: 32 # = 2×r usually
  target_modules: ["q_proj","v_proj","k_proj","o_proj"]
  dropout: 0.05

During training:
  ✓ is train_loss decreasing?
  ✓ val_loss is not growing? (if growing → overfitting)
  ✓ Is grad_norm stable? (spike → lr too high)

After training:
  ✓ target_metric on test set vs baseline
  ✓ general benchmarks have not fallen?
     (MT-Bench, MMLU, HumanEval)
  ✓ human evaluation 50 examples
  ✓ compare vs prompted GPT-4 on the same task

1. Have you exhausted your prompting?
   Have you tried few-shot, CoT, persona, chain?
   No → prompting first, it's free

2. Do you have data and metrics?
   Minimum 200 quality examples?
   A clear metric by which to measure improvement?
   No → first collect data and define a metric

3. Is the problem in behavior, not knowledge?
   If you add the correct data to the prompt → does everything work?
   Yes → use RAG, not fine tuning

All three yes? Fine-tune.
Start with LoRA on 7-8B models.
Compare with prompted GPT-4 on your task.