BERT, Encoders & Non-Generative Models

┌────────────────────────────────── ───────────────────────────────────┐
│ ENCODER-ONLY ENCODER-DECODER DECODER-ONLY │
│ BERT, RoBERTa T5, BART, mT5 GPT, Llama │
│ DeBERTa, XLM-R FLAN-T5, mBART Claude, Mistral │
├────────────────────────────────── ───────────────────────────────────┤
│ Sees text: Sees text: Sees text: │
│ BIDIRECTIONAL input - bidirectional.         ONLY on the left │
│ t₀↔t₁↔t₂↔t₃ exit - left→right t₀→t₁→t₂→t₃ │
├────────────────────────────────── ───────────────────────────────────┤
│ Learning Objective: Learning Objective: Learning Objective: │
│ Restore Transform Predict │
│ masked tokens input to output next token │
├────────────────────────────────── ───────────────────────────────────┤
│ Strengths: Strengths: Strengths: │
│ Understanding, embeddings Structural generation Free gen.  │
│ Classification, NER Translation, summarization Reasoning │
│ Reranking, similarity Conditional generation Few-shot, RAG │
├────────────────────────────────── ───────────────────────────────────┤
│ Weaknesses: Weaknesses: Weaknesses: │
│ Does not generate text More difficult in deployment Worse to understand │
│ Fixed inlet length.   More expensive encoder-only separate token │
└────────────────────────────────── ───────────────────────────────────┘

Understanding (encoder):
  input: complete sentence
  attention: each token sees ALL others
  output: enriched vectors of each token
  → perfect for “what does this word mean here?”

Generation (decoder):
  input: everything that has happened so far
  attention: causal mask - only the past
  output: next token
  → perfect for “what’s coming next?”

Analogy:
  Encoder = editor: reads the entire text,
            understands every word in the context of everything

  Decoder = writer: writes word by word,
            each word knows only the previous ones

What everyone has in common:
  ✓ Architecture: self-attention + MLP + residuals
  ✓ Training: gradient descent + backprop
  ✓ Tokenization: BPE or WordPiece
  ✓ Pretraining on a large text corpus

How are they different:
  ↔ attention direction (uni vs bidirectional)
  ↔ pre-training task (MLM vs CLM)
  ↔ size (110M vs 70B+ parameters)
  ↔ specialization (understanding vs generation)

All of them are neural networks, all of them are transformers.
"LLM" = large decoder, nothing more.

Login (original):
  "The cat is sitting on a warm windowsill"

Login for the model (with masks - 15% random tokens):
  "Cat [MASK] on the [MASK] windowsill"

The model sees:
  "Cat" ←→ "[MASK]" ←→ "on" ←→ "[MASK]" ←→ "windowsill"
  ↑ BIDIRECTIONAL attention: each token sees everything

Task: predict masked tokens
  [MASK]₁ → softmax → P("sitting")=0.71, P("lying")=0.18, …
  [MASK]₂ → softmax → P("warm")=0.54, P("cold")=0.23, …

Loss: cross-entropy only on masked tokens

Why is it smarter than CLM (predict next token):
  CLM: "The cat is sitting" → predict "on" ← sees only the left context
  MLM: "Cat [M] on [M] window sill" ← sees BOTH sides!
  → to guess [MASK]₁, you need to take into account both “cat” and “windowsill”
  → forces the model to understand syntax and semantics more deeply

BERT (bidirectional - entire matrix):

       k o t s i d i t
  to [✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓]
  o [✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓]
  t [✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓]
  ↑ "cat" knows "sits" during processing - from the future

GPT (causal - lower triangle):

       k o t s i d i t
  to [✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗]
  o [✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗]
  t [✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗]
  ↑ "cat" does not know what will happen "sits" - only the past

Entry format (one sentence):
  [CLS] Cat sitting on the windowsill [SEP]

Entry format (a couple of sentences):
  [CLS] Question [SEP] Answer [SEP]
  [CLS] Text A [SEP] Text B [SEP]

NSP - pre-training task:
  50%: [CLS] The cat is sitting. [SEP] He's watching. →IsNext
  50%: [CLS] The cat is sitting. [SEP] Borscht recipe. →NotNext
  predict: IsNext / NotNext from h[CLS]

Using [CLS] for fine-tuning:
  h_cls = bert_output[0] ← vector [CLS] token
  logits = Linear(h_cls, num_classes)
  → classification of the entire proposal

RoBERTa: NSP removed → quality increased

# Frequent words - one token:
"cat" → [cat]
"sits" → [sits]

# Rare words are broken:
"neural network" → [neuro, ##network]
"window sill" → [under, ##con, ##nick]
"tokenization" → [token, ##ization]

# Special tokens:
[CLS] → id: 101 ← beginning, aggregation of meaning
[SEP] → id: 102 ← sentence separator
[MASK]→ id: 103 ← mask for MLM
[UNK] → id: 100 ← unknown token

Vocab size BERT: ~30k (vs GPT-4: ~100k)
→ more fractional tokenization, but morphemic

Removed NSP:
  → BERT was trained to predict the “next sentence”
  → it turned out that this does not help downstream tasks
  → instead: pack full offers to 512 tokens

Dynamic Masking:
  → BERT: masks are fixed during preprocessing
  → RoBERTa: new masks every era
  → the model sees each sentence with different masks

Scale:
  BERT: 16GB of text, 40 epochs, batch 256
  RoBERTa: 160GB of text, batch 8192, longer training

Result: +2-5% on GLUE benchmark

Normal attention:
  score(i,j) = (H_i + P_i) · (H_j + P_j)ᵀ
  ↑ content and position mixed

DeBERTa disentangled:
  score(i,j) = H_i·H_jᵀ ← content-to-content
             + H_i·P_{i→j}ᵀ ← content-to-position
             + P_{i→j}·H_jᵀ ← position-to-content
  ↑ relative positions instead of absolute ones

Enhanced Mask Decoder (EMD):
  → for MLM: a special layer returns
     absolute positions before prediction
  → better at guessing disguised tokens

DeBERTa-v3: the best encoder for GLUE/SuperGLUE (2021-2023)

Training:
  2.5TB of text in 100 languages (Common Crawl)
  vocab: 250k tokens (sentencepiece, multilingual BPE)
  → BERT: 30k monolingual. | XLM-R: 250k multilingual

Zero-shot transfer:
  fine-tune on NER (English)
  → apply in Russian/Chinese/Arabic without additional data
  → works because tokens are in different languages
     have similar embeddings if they mean the same thing

Dimensions:
  xlm-roberta-base: 278M parameters
  xlm-roberta-large: 560M parameters

For Russian NLP: standard baseline

Architectural improvements:
  ✓ RoPE instead of learned positional embeddings
  ✓ Flash Attention 2 → faster + long contexts
  ✓ GeGLU instead of GELU activation
  ✓ Pre-norm (RMSNorm) as in Llama
  ✓ Context up to 8192 tokens (BERT: 512)
  ✓Alternating local+global attention

Data:
  2T tokens (text + code)
  → understands code - unique to encoder

Dimensions:
  modernbert-base: 149M
  modernbert-large: 395M

→ the best choice for new projects (2025)

①SEQUENCE CLASSIFICATION (intent, sentiment, topic):

  Input: "[CLS] What is the shipping price? [SEP]"
         ↓ BERT encoder (12 layers)
  h_cls ∈ ℝ^768 ← vector [CLS]
         ↓ Linear(768 → num_classes) + softmax
  P(shipping_query)=0.94, P(price_query)=0.04, …

② TOKEN CLASSIFICATION (NER, POS-tagging):

  Input: "[CLS] Ivan Petrov bought a MacBook [SEP]"
         ↓BERT encoder
  h_0 h_1 h_2 h_3 h_4 h_5 ← vector of each token
         ↓ Linear(768 → num_tags) for each position
  O B-PER I-PER O B-PROD O

③ SENTENCE-PAIR (similarity, NLI, reranking):

  Input: "[CLS] request [SEP] document [SEP]"
         ↓ BERT encoder
  h_cls ∈ ℝ^768
         ↓ Linear(768 → 3) [entailment / neutral / contradiction]
  attraction=0.91 ← relevant!

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
  "roberta-base",
  num_labels=5
)

Recommended hyperparameters:
  lr: 2e-5 – 5e-5 ← low! don't break pretrain
  batch_size: 16 – 32
  epochs: 3 – 5 ← retraining quickly
  warmup: 6% steps
  weight_decay: 0.01

Data:
  100+ examples → works
  1000+ examples → good
  10k+ examples → excellent

Teacher: BERT-large (340M), accurate
Student: DistilBERT (66M), fast

Student Loss = combination of three goals:
① Distillation loss:
  KL(softmax(logits_student / T) ‖ softmax(logits_teacher / T))
  T=temperature→ soft probabilities = “dark knowledge”

② Cosine embedding loss:
  hidden states of student ≈ teacher

③ MLM loss:
  standard training on masked tokens

Popular distilled models:
  DistilBERT: 66M, 60% faster, 97% BERT quality
  TinyBERT: 15M, 9.4× faster, 96% quality
  MobileBERT: 25M, edge deployment

Architecture (example: translation):

  Entry: "The cat sat on the mat"
              ↓
  ┌─────────────────── ────────────────────┐
  │ ENCODER (bidirectional self-attention)│
  │ h₁ h₂ h₃ h₄ h₅ h₆ │
  │ The cat sat on the mat │
  └─────────────────── ┬───────────────────┘
                      │ cross-attention
                      ↓ (decoder looks at ALL h encoder)
  ┌─────────────────── ────────────────────┐
  │ DECODER (causal + cross-attention) │
  │ [START] → Cat → sat → on → … │
  │ │
  │ At every step: │
  │ 1. Causal self-attn (past tokens) │
  │ 2. Cross-attention to encoder h₁..h₆ │
  │ 3. FFN → logits → next token │
  └─────────────────── ────────────────────┘

Why is it better than decoder-only for translation:
  → encoder reads the entire input phrase ENTIRELY
  → decoder knows when generating each word
     ALL source text via cross-attention
  → less “forgetting” of long inputs

# All tasks are in the same format:
"translate English to Russian: Hello world"
  → "Hello world"

"summarize: [long text...]"
  → "Summary: ..."

"sst2 sentence: I love this movie"
  → "positive"

"cola sentence: The cat sat"
  → "acceptable"

FLAN-T5 (2022):
  → T5 + instruction tuning on 1000+ tasks
  → follows instructions like LLM, but less
  → flan-t5-large (780M) vs GPT-3 (175B)
  → wins on structured generation

Original: "The cat sat on the warm windowsill and watched"

Token masking: "The cat [M] was on the [M] windowsill watching"
Token deletion: "The cat was watching on the warm windowsill"
Text infilling: "The cat [M M M] was looking" ← one [M] = many words
Sentence permut: "I looked. The cat was sitting on the windowsill and"
Document rotation: start from the middle

Why does this help summarization:
  → infilling = generating a missing fragment
  → summarization = generation of a “compressed” fragment
  → structurally similar tasks

mBART: multilingual BART, 25 languages, including ru

Real numbers (2025, typical production):

Model Latency Cost/1M Accuracy Parameters
───────────────────────────── ──────────────────────────────
fine-tuned roberta 3ms $0.01 96% 125M
fine-tuned deberta 8ms $0.02 97.5% 184M
claude-haiku-3-5 300ms $0.80 91% ~20B?
gpt-4o-mini 400ms $0.60 93% ~20B?
claude-sonnet-4-6 1200ms $15.00 99% ~200B?
gpt-4o 1500ms $10.00 98% ~200B?
───────────────────────────── ──────────────────────────────
* accuracy for the intent classification task with 5000 examples

Conclusion: fine-tuned RoBERTa is 100x cheaper and 400x faster
GPT-4 gives only ~3% increase in accuracy.
At 1M requests/day: $10 vs $10,000 per day.

When GPT-4 is needed:
  → the task requires reasoning through several steps
  → no marked data for fine-tuning
  → the task changes every day
  → need a detailed text answer

① Classification of intents (IntentRouter):
  fine-tuned RoBERTa: 96.5% F1, 3ms
  claude-haiku prompt: 91.2% F1, 280ms
  → 100x faster, more precisely, 80x cheaper

② NER (Entity Extraction):
  fine-tuned BERT/DeBERTa: F1=0.92
  GPT-4 prompting: F1=0.87
  → encoder is more accurate on clearly labeled data

③ Reranking in RAG:
  cross-encoder reranker: NDCG@10=0.74
  embedding cosine: NDCG@10=0.61
  → cross-encoder sees query+doc together

④ Semantic similarity (deduplication):
  bi-encoder (E5): 50k par/sec
  LLM: 10 pairs/sec
  → scale is not possible with LLM

⑤ Content moderation:
  fine-tuned toxicity classifier: 0.5ms/request
  → should be cheaper than the LLM request itself

User request
       ↓
[RoBERTa IntentClassifier] 3ms, $0.00001
  intent="price_query" (conf=0.96) → PriceAgent
  intent="complaint" (conf=0.91) → ComplaintAgent
  intent="unclear" (conf=0.43) → ↓ (LLM needed)
                                              ↓
                             [Claude Haiku] 280ms, $0.001
                              clarification?
                              simple_task? → answer directly
                              complex?    → ↓
                                           [Claude Sonnet] 1200ms, $0.015

Typical traffic statistics:
  70% → RoBERTa copes ($0.00001/request)
  25% → Haiku ($0.001/request)
   5% → Sonnet ($0.015/request)
  Average cost: ~$0.002 (vs $0.015 if everything is on Sonnet)

Need to GENERATE text?
  No → continue below
  Yes → fixed structure A→B? (translation, summarization)
          Yes → Encoder-Decoder: T5, FLAN-T5, BART, mBART
          No → free generation, reasoning, dialogue
               → Decoder-only LLM: Claude, GPT-4o, Llama-3

────────────────────────── ───────────────────────────

Task type (without generation):

  Text classification (intent, sentiment, topic, toxicity)
    Are there 500+ examples?
      Yes → fine-tuned RoBERTa / DeBERTa / ModernBERT
      No → zero-shot Claude Haiku / GPT-4o-mini

  Entity extraction (NER, spans)
    Are there 200+ marked up offers?
      Yes → fine-tuned BERT / RoBERTa (token classification)
      No -> LLM with JSON output

  Semantic embeddings (search, clustering, deduplication)
    Do you need Multilingual?
      Yes → multilingual-e5, paraphrase-multilingual-mpnet
      No → E5-large-v2, BGE-large-en, nomic-embed-text

  Reranking (clarification of top-K RAG results)
    → cross-encoder: bge-reranker-v2, ms-marco-MiniLM

  Sentence similarity/Textual Entailment
    → fine-tuned DeBERTa-NLI, SBERT

  Multilingual classification
    → fine-tuned XLM-RoBERTa