BERT, Encoders & Non-Generative Models

Understanding (encoder):
  input: complete sentence
  attention: each token sees ALL others
  output: enriched vectors of each token
  → perfect for “what does this word mean here?”

Generation (decoder):
  input: everything that has happened so far
  attention: causal mask - only the past
  output: next token
  → perfect for “what’s coming next?”

Analogy:
  Encoder = editor: reads the entire text,
            understands every word in the context of everything

  Decoder = writer: writes word by word,
            each word knows only the previous ones

What everyone has in common:
  ✓ Architecture: self-attention + MLP + residuals
  ✓ Training: gradient descent + backprop
  ✓ Tokenization: BPE or WordPiece
  ✓ Pretraining on a large text corpus

How are they different:
  ↔ attention direction (uni vs bidirectional)
  ↔ pre-training task (MLM vs CLM)
  ↔ size (110M vs 70B+ parameters)
  ↔ specialization (understanding vs generation)

All of them are neural networks, all of them are transformers.
"LLM" = large decoder, nothing more.

Entry format (one sentence):
  [CLS] Cat sitting on the windowsill [SEP]

Entry format (a couple of sentences):
  [CLS] Question [SEP] Answer [SEP]
  [CLS] Text A [SEP] Text B [SEP]

NSP - pre-training task:
  50%: [CLS] The cat is sitting. [SEP] He's watching. →IsNext
  50%: [CLS] The cat is sitting. [SEP] Borscht recipe. →NotNext
  predict: IsNext / NotNext from h[CLS]

Using [CLS] for fine-tuning:
  h_cls = bert_output[0] ← vector [CLS] token
  logits = Linear(h_cls, num_classes)
  → classification of the entire proposal

RoBERTa: NSP removed → quality increased

# Frequent words - one token:
"cat" → [cat]
"sits" → [sits]

# Rare words are broken:
"neural network" → [neuro, ##network]
"window sill" → [under, ##con, ##nick]
"tokenization" → [token, ##ization]

# Special tokens:
[CLS] → id: 101 ← beginning, aggregation of meaning
[SEP] → id: 102 ← sentence separator
[MASK]→ id: 103 ← mask for MLM
[UNK] → id: 100 ← unknown token

Vocab size BERT: ~30k (vs GPT-4: ~100k)
→ more fractional tokenization, but morphemic

Removed NSP:
  → BERT was trained to predict the “next sentence”
  → it turned out that this does not help downstream tasks
  → instead: pack full offers to 512 tokens

Dynamic Masking:
  → BERT: masks are fixed during preprocessing
  → RoBERTa: new masks every era
  → the model sees each sentence with different masks

Scale:
  BERT: 16GB of text, 40 epochs, batch 256
  RoBERTa: 160GB of text, batch 8192, longer training

Result: +2-5% on GLUE benchmark

Normal attention:
  score(i,j) = (H_i + P_i) · (H_j + P_j)ᵀ
  ↑ content and position mixed

DeBERTa disentangled:
  score(i,j) = H_i·H_jᵀ ← content-to-content
             + H_i·P_{i→j}ᵀ ← content-to-position
             + P_{i→j}·H_jᵀ ← position-to-content
  ↑ relative positions instead of absolute ones

Enhanced Mask Decoder (EMD):
  → for MLM: a special layer returns
     absolute positions before prediction
  → better at guessing disguised tokens

DeBERTa-v3: the best encoder for GLUE/SuperGLUE (2021-2023)

Training:
  2.5TB of text in 100 languages (Common Crawl)
  vocab: 250k tokens (sentencepiece, multilingual BPE)
  → BERT: 30k monolingual. | XLM-R: 250k multilingual

Zero-shot transfer:
  fine-tune on NER (English)
  → apply in Russian/Chinese/Arabic without additional data
  → works because tokens are in different languages
     have similar embeddings if they mean the same thing

Dimensions:
  xlm-roberta-base: 278M parameters
  xlm-roberta-large: 560M parameters

For Russian NLP: standard baseline

Architectural improvements:
  ✓ RoPE instead of learned positional embeddings
  ✓ Flash Attention 2 → faster + long contexts
  ✓ GeGLU instead of GELU activation
  ✓ Pre-norm (RMSNorm) as in Llama
  ✓ Context up to 8192 tokens (BERT: 512)
  ✓Alternating local+global attention

Data:
  2T tokens (text + code)
  → understands code - unique to encoder

Dimensions:
  modernbert-base: 149M
  modernbert-large: 395M

→ the best choice for new projects (2025)

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
  "roberta-base",
  num_labels=5
)

Recommended hyperparameters:
  lr: 2e-5 – 5e-5 ← low! don't break pretrain
  batch_size: 16 – 32
  epochs: 3 – 5 ← retraining quickly
  warmup: 6% steps
  weight_decay: 0.01

Data:
  100+ examples → works
  1000+ examples → good
  10k+ examples → excellent

Teacher: BERT-large (340M), accurate
Student: DistilBERT (66M), fast

Student Loss = combination of three goals:
① Distillation loss:
  KL(softmax(logits_student / T) ‖ softmax(logits_teacher / T))
  T=temperature→ soft probabilities = “dark knowledge”

② Cosine embedding loss:
  hidden states of student ≈ teacher

③ MLM loss:
  standard training on masked tokens

Popular distilled models:
  DistilBERT: 66M, 60% faster, 97% BERT quality
  TinyBERT: 15M, 9.4× faster, 96% quality
  MobileBERT: 25M, edge deployment

# All tasks are in the same format:
"translate English to Russian: Hello world"
  → "Hello world"

"summarize: [long text...]"
  → "Summary: ..."

"sst2 sentence: I love this movie"
  → "positive"

"cola sentence: The cat sat"
  → "acceptable"

FLAN-T5 (2022):
  → T5 + instruction tuning on 1000+ tasks
  → follows instructions like LLM, but less
  → flan-t5-large (780M) vs GPT-3 (175B)
  → wins on structured generation

Original: "The cat sat on the warm windowsill and watched"

Token masking: "The cat [M] was on the [M] windowsill watching"
Token deletion: "The cat was watching on the warm windowsill"
Text infilling: "The cat [M M M] was looking" ← one [M] = many words
Sentence permut: "I looked. The cat was sitting on the windowsill and"
Document rotation: start from the middle

Why does this help summarization:
  → infilling = generating a missing fragment
  → summarization = generation of a “compressed” fragment
  → structurally similar tasks

mBART: multilingual BART, 25 languages, including ru

① Classification of intents (IntentRouter):
  fine-tuned RoBERTa: 96.5% F1, 3ms
  claude-haiku prompt: 91.2% F1, 280ms
  → 100x faster, more precisely, 80x cheaper

② NER (Entity Extraction):
  fine-tuned BERT/DeBERTa: F1=0.92
  GPT-4 prompting: F1=0.87
  → encoder is more accurate on clearly labeled data

③ Reranking in RAG:
  cross-encoder reranker: NDCG@10=0.74
  embedding cosine: NDCG@10=0.61
  → cross-encoder sees query+doc together

④ Semantic similarity (deduplication):
  bi-encoder (E5): 50k par/sec
  LLM: 10 pairs/sec
  → scale is not possible with LLM

⑤ Content moderation:
  fine-tuned toxicity classifier: 0.5ms/request
  → should be cheaper than the LLM request itself