{
  "type": "new-runtime-knowledge-guide",
  "version": 1,
  "generated_at": "2026-07-23",
  "volume": 10,
  "slug": "bert-encoder-reference",
  "title": "BERT, Encoders & Non-Generative Models",
  "description": "Taxonomy of transformer architectures: encoder-only, decoder-only, encoder-decoder. How BERT works and why bidirectional is better to understand. BERT family. Fine-tuning patterns. T5 and seq2seq. When a small specialized model beats GPT-4. A practical map for choosing an architecture for a task.",
  "track": {
    "slug": "ml-and-decision-systems",
    "title": "ML & Decision Systems",
    "route": "https://newruntime.com/learn/tracks/ml-and-decision-systems/",
    "position": 2
  },
  "counts": {
    "sections": 7,
    "cards": 23
  },
  "routes": {
    "html": "https://newruntime.com/learn/bert-encoder-reference/",
    "json": "https://newruntime.com/learn/bert-encoder-reference.json"
  },
  "sections": [
    {
      "number": "01",
      "title": "Taxonomy: three families of architectures",
      "intro": "",
      "cards": [
        {
          "number": "01",
          "title": "Three families of transformers: what each one can do",
          "tag": "Taxonomy",
          "description": "All modern NLP models are built on transformers, but they are designed fundamentally differently. The difference is not in “power” - but in how the model sees the text and what it can do with this knowledge. The choice of architecture is the first and most important engineering choice.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "┌────────────────────────────────── ───────────────────────────────────┐\n│ ENCODER-ONLY ENCODER-DECODER DECODER-ONLY │\n│ BERT, RoBERTa T5, BART, mT5 GPT, Llama │\n│ DeBERTa, XLM-R FLAN-T5, mBART Claude, Mistral │\n├────────────────────────────────── ───────────────────────────────────┤\n│ Sees text: Sees text: Sees text: │\n│ BIDIRECTIONAL input - bidirectional.         ONLY on the left │\n│ t₀↔t₁↔t₂↔t₃ exit - left→right t₀→t₁→t₂→t₃ │\n├────────────────────────────────── ───────────────────────────────────┤\n│ Learning Objective: Learning Objective: Learning Objective: │\n│ Restore Transform Predict │\n│ masked tokens input to output next token │\n├────────────────────────────────── ───────────────────────────────────┤\n│ Strengths: Strengths: Strengths: │\n│ Understanding, embeddings Structural generation Free gen.  │\n│ Classification, NER Translation, summarization Reasoning │\n│ Reranking, similarity Conditional generation Few-shot, RAG │\n├────────────────────────────────── ───────────────────────────────────┤\n│ Weaknesses: Weaknesses: Weaknesses: │\n│ Does not generate text More difficult in deployment Worse to understand │\n│ Fixed inlet length.   More expensive encoder-only separate token │\n└────────────────────────────────── ───────────────────────────────────┘"
            },
            {
              "kind": "list",
              "label": "key selection principle",
              "items": [
                "understand text → encoder-only",
                "convert A→B→encoder-decoder",
                "generate freely → decoder-only",
                "LLM can do all three - but it’s expensive and slow"
              ]
            }
          ]
        },
        {
          "number": "02",
          "title": "Understanding vs Generation: Different Tasks",
          "tag": "Taxonomy",
          "description": "“Understand” and “generate” are fundamentally different operations, and the architectures are optimized for each. Encoder reads everything at once and builds a deep understanding. Decoder only reads the past and predicts the future.",
          "blocks": [
            {
              "kind": "code",
              "label": "operations and architecture",
              "language": "text",
              "value": "Understanding (encoder):\n  input: complete sentence\n  attention: each token sees ALL others\n  output: enriched vectors of each token\n  → perfect for “what does this word mean here?”\n\nGeneration (decoder):\n  input: everything that has happened so far\n  attention: causal mask - only the past\n  output: next token\n  → perfect for “what’s coming next?”\n\nAnalogy:\n  Encoder = editor: reads the entire text,\n            understands every word in the context of everything\n\n  Decoder = writer: writes word by word,\n            each word knows only the previous ones"
            }
          ]
        },
        {
          "number": "03",
          "title": "What it does “not LLM” mean: small vs large",
          "tag": "Taxonomy",
          "description": "\"LLM\" is an informal term for large decoder-only models. Encoder models are not LLMs not because they are bad, but because they solve other problems. BERT-base (110M parameters) vs GPT-4 (~1T) is not a “small LLM”, it is a different tool.",
          "blocks": [
            {
              "kind": "code",
              "label": "what unites all transformer models",
              "language": "text",
              "value": "What everyone has in common:\n  ✓ Architecture: self-attention + MLP + residuals\n  ✓ Training: gradient descent + backprop\n  ✓ Tokenization: BPE or WordPiece\n  ✓ Pretraining on a large text corpus\n\nHow are they different:\n  ↔ attention direction (uni vs bidirectional)\n  ↔ pre-training task (MLM vs CLM)\n  ↔ size (110M vs 70B+ parameters)\n  ↔ specialization (understanding vs generation)\n\nAll of them are neural networks, all of them are transformers.\n\"LLM\" = large decoder, nothing more."
            }
          ]
        }
      ]
    },
    {
      "number": "02",
      "title": "How BERT works",
      "intro": "",
      "cards": [
        {
          "number": "04",
          "title": "MLM: Masked Language Modeling - BERT pre-training task",
          "tag": "BERT",
          "description": "BERT is trained on the “recover a masked token” task. 15% of the tokens are randomly masked, and the model learns to predict them from the remaining ones. Because the model sees context from both sides, it is forced to learn to deeply understand the meaning of words.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "Login (original):\n  \"The cat is sitting on a warm windowsill\"\n\nLogin for the model (with masks - 15% random tokens):\n  \"Cat [MASK] on the [MASK] windowsill\"\n\nThe model sees:\n  \"Cat\" ←→ \"[MASK]\" ←→ \"on\" ←→ \"[MASK]\" ←→ \"windowsill\"\n  ↑ BIDIRECTIONAL attention: each token sees everything\n\nTask: predict masked tokens\n  [MASK]₁ → softmax → P(\"sitting\")=0.71, P(\"lying\")=0.18, …\n  [MASK]₂ → softmax → P(\"warm\")=0.54, P(\"cold\")=0.23, …\n\nLoss: cross-entropy only on masked tokens\n\nWhy is it smarter than CLM (predict next token):\n  CLM: \"The cat is sitting\" → predict \"on\" ← sees only the left context\n  MLM: \"Cat [M] on [M] window sill\" ← sees BOTH sides!\n  → to guess [MASK]₁, you need to take into account both “cat” and “windowsill”\n  → forces the model to understand syntax and semantics more deeply"
            },
            {
              "kind": "list",
              "label": "masking details (original BERT)",
              "items": [
                "80% → [MASK] is replaced by a mask",
                "10% → replaced with a random token",
                "10% → remains as is",
                "why not 100% masks: the model learns to “expect” regular tokens too"
              ]
            }
          ]
        },
        {
          "number": "05",
          "title": "Bidirectional vs Causal Attention",
          "tag": "BERT",
          "description": "The main difference between BERT and GPT is the direction of attention. In BERT, each token can “look” at all others, including future ones. In GPT - only for past ones. This makes BERT better at understanding but unable to generate.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "sql",
              "value": "BERT (bidirectional - entire matrix):\n\n       k o t s i d i t\n  to [✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓]\n  o [✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓]\n  t [✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓]\n  ↑ \"cat\" knows \"sits\" during processing - from the future\n\nGPT (causal - lower triangle):\n\n       k o t s i d i t\n  to [✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗]\n  o [✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗]\n  t [✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗]\n  ↑ \"cat\" does not know what will happen \"sits\" - only the past"
            }
          ]
        },
        {
          "number": "06",
          "title": "[CLS], [SEP] and the second NSP pretraining object",
          "tag": "BERT",
          "description": "[CLS] is a special token at the beginning of each BERT input. After training, its vector aggregates the meaning of the entire sentence. NSP (Next Sentence Prediction) was the second learning object that forced [CLS] to understand the relationships between pairs of sentences.",
          "blocks": [
            {
              "kind": "code",
              "label": "input format and usage",
              "language": "sql",
              "value": "Entry format (one sentence):\n  [CLS] Cat sitting on the windowsill [SEP]\n\nEntry format (a couple of sentences):\n  [CLS] Question [SEP] Answer [SEP]\n  [CLS] Text A [SEP] Text B [SEP]\n\nNSP - pre-training task:\n  50%: [CLS] The cat is sitting. [SEP] He's watching. →IsNext\n  50%: [CLS] The cat is sitting. [SEP] Borscht recipe. →NotNext\n  predict: IsNext / NotNext from h[CLS]\n\nUsing [CLS] for fine-tuning:\n  h_cls = bert_output[0] ← vector [CLS] token\n  logits = Linear(h_cls, num_classes)\n  → classification of the entire proposal\n\nRoBERTa: NSP removed → quality increased"
            }
          ]
        },
        {
          "number": "07",
          "title": "WordPiece: BERT tokenization",
          "tag": "BERT",
          "description": "BERT uses WordPiece, a variant of BPE. Rare words are broken into parts with the prefix “##”. This allows you to process unfamiliar words through morphemes and keep a dictionary of a reasonable size (~30k tokens).",
          "blocks": [
            {
              "kind": "code",
              "label": "tokenization examples",
              "language": "text",
              "value": "# Frequent words - one token:\n\"cat\" → [cat]\n\"sits\" → [sits]\n\n# Rare words are broken:\n\"neural network\" → [neuro, ##network]\n\"window sill\" → [under, ##con, ##nick]\n\"tokenization\" → [token, ##ization]\n\n# Special tokens:\n[CLS] → id: 101 ← beginning, aggregation of meaning\n[SEP] → id: 102 ← sentence separator\n[MASK]→ id: 103 ← mask for MLM\n[UNK] → id: 100 ← unknown token\n\nVocab size BERT: ~30k (vs GPT-4: ~100k)\n→ more fractional tokenization, but morphemic"
            }
          ]
        }
      ]
    },
    {
      "number": "03",
      "title": "BERT family: evolution and variants",
      "intro": "",
      "cards": [
        {
          "number": "08",
          "title": "RoBERTa: BERT without NSP and with more data",
          "tag": "Family",
          "description": "RoBERTa (2019, Meta) - retrained BERT with better hyperparameters. Three key changes: removed NSP (turned out to be useless), increased the batch by 8 times, trained 10x longer on a larger corpus. The result is a significant increase in all benchmarks.",
          "blocks": [
            {
              "kind": "code",
              "label": "what changed vs BERT",
              "language": "text",
              "value": "Removed NSP:\n  → BERT was trained to predict the “next sentence”\n  → it turned out that this does not help downstream tasks\n  → instead: pack full offers to 512 tokens\n\nDynamic Masking:\n  → BERT: masks are fixed during preprocessing\n  → RoBERTa: new masks every era\n  → the model sees each sentence with different masks\n\nScale:\n  BERT: 16GB of text, 40 epochs, batch 256\n  RoBERTa: 160GB of text, batch 8192, longer training\n\nResult: +2-5% on GLUE benchmark"
            },
            {
              "kind": "list",
              "label": "when to choose RoBERTa vs BERT",
              "items": [
                "RoBERTa → almost always better than BERT",
                "roberta-base: 125M → fast",
                "roberta-large: 355M → more precisely"
              ]
            }
          ]
        },
        {
          "number": "09",
          "title": "DeBERTa: disentangled attention",
          "tag": "Family",
          "description": "DeBERTa (2021, Microsoft) - the best encoder-only for English. Key idea: separate the content of the token and its position in attention. Each pair of tokens has two attention scores: content-to-content and position-to-content.",
          "blocks": [
            {
              "kind": "code",
              "label": "disentangled attention and enhanced mask decoder",
              "language": "text",
              "value": "Normal attention:\n  score(i,j) = (H_i + P_i) · (H_j + P_j)ᵀ\n  ↑ content and position mixed\n\nDeBERTa disentangled:\n  score(i,j) = H_i·H_jᵀ ← content-to-content\n             + H_i·P_{i→j}ᵀ ← content-to-position\n             + P_{i→j}·H_jᵀ ← position-to-content\n  ↑ relative positions instead of absolute ones\n\nEnhanced Mask Decoder (EMD):\n  → for MLM: a special layer returns\n     absolute positions before prediction\n  → better at guessing disguised tokens\n\nDeBERTa-v3: the best encoder for GLUE/SuperGLUE (2021-2023)"
            }
          ]
        },
        {
          "number": "10",
          "title": "XLM-R: multilingual encoder",
          "tag": "Family",
          "description": "XLM-RoBERTa (2020, Meta) - RoBERTa trained in 100 languages simultaneously. The only model understands Russian, English, Chinese and 97 other languages. Zero-shot transfer: train on English data, apply on Russian.",
          "blocks": [
            {
              "kind": "code",
              "label": "key properties",
              "language": "text",
              "value": "Training:\n  2.5TB of text in 100 languages (Common Crawl)\n  vocab: 250k tokens (sentencepiece, multilingual BPE)\n  → BERT: 30k monolingual. | XLM-R: 250k multilingual\n\nZero-shot transfer:\n  fine-tune on NER (English)\n  → apply in Russian/Chinese/Arabic without additional data\n  → works because tokens are in different languages\n     have similar embeddings if they mean the same thing\n\nDimensions:\n  xlm-roberta-base: 278M parameters\n  xlm-roberta-large: 560M parameters\n\nFor Russian NLP: standard baseline"
            }
          ]
        },
        {
          "number": "11",
          "title": "ModernBERT: BERT in 2024",
          "tag": "Family",
          "description": "ModernBERT (late 2024, Answer.AI + LightOn) - a complete rethinking of the encoder architecture using everything we learned in 6 years after BERT: RoPE, Flash Attention, SwiGLU, deep learning on 2T tokens. Beats DeBERTa at smaller sizes.",
          "blocks": [
            {
              "kind": "code",
              "label": "what's new vs classic BERT",
              "language": "text",
              "value": "Architectural improvements:\n  ✓ RoPE instead of learned positional embeddings\n  ✓ Flash Attention 2 → faster + long contexts\n  ✓ GeGLU instead of GELU activation\n  ✓ Pre-norm (RMSNorm) as in Llama\n  ✓ Context up to 8192 tokens (BERT: 512)\n  ✓Alternating local+global attention\n\nData:\n  2T tokens (text + code)\n  → understands code - unique to encoder\n\nDimensions:\n  modernbert-base: 149M\n  modernbert-large: 395M\n\n→ the best choice for new projects (2025)"
            }
          ]
        }
      ]
    },
    {
      "number": "04",
      "title": "Fine-tuning patterns for Encoder models",
      "intro": "",
      "cards": [
        {
          "number": "12",
          "title": "Three main fine-tuning patterns",
          "tag": "Fine-tuning",
          "description": "Encoder models are universal “understanders” of text. A specific problem is solved by adding a small “head” on top of frozen or additionally trained scales. Three patterns cover 90% of real text problems.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "①SEQUENCE CLASSIFICATION (intent, sentiment, topic):\n\n  Input: \"[CLS] What is the shipping price? [SEP]\"\n         ↓ BERT encoder (12 layers)\n  h_cls ∈ ℝ^768 ← vector [CLS]\n         ↓ Linear(768 → num_classes) + softmax\n  P(shipping_query)=0.94, P(price_query)=0.04, …\n\n② TOKEN CLASSIFICATION (NER, POS-tagging):\n\n  Input: \"[CLS] Ivan Petrov bought a MacBook [SEP]\"\n         ↓BERT encoder\n  h_0 h_1 h_2 h_3 h_4 h_5 ← vector of each token\n         ↓ Linear(768 → num_tags) for each position\n  O B-PER I-PER O B-PROD O\n\n③ SENTENCE-PAIR (similarity, NLI, reranking):\n\n  Input: \"[CLS] request [SEP] document [SEP]\"\n         ↓ BERT encoder\n  h_cls ∈ ℝ^768\n         ↓ Linear(768 → 3) [entailment / neutral / contradiction]\n  attraction=0.91 ← relevant!"
            },
            {
              "kind": "list",
              "label": "which pattern for which task",
              "items": [
                "intent/tonality → sequence classification",
                "NER, entity extraction → token classification",
                "reranking, fact checking → sentence-pair",
                "embeddings → no head, take h_cls or mean"
              ]
            }
          ]
        },
        {
          "number": "13",
          "title": "Fine-tuning practice: hyperparameters",
          "tag": "Fine-tuning",
          "description": "Fine-tuning an encoder model requires much less data and calculations than LLM training. 1000 examples are often enough for a good classification. The main danger is catastrophic forgetting when lr is too high.",
          "blocks": [
            {
              "kind": "code",
              "label": "typical hyperparameters",
              "language": "sql",
              "value": "from transformers import AutoModelForSequenceClassification\n\nmodel = AutoModelForSequenceClassification.from_pretrained(\n  \"roberta-base\",\n  num_labels=5\n)\n\nRecommended hyperparameters:\n  lr: 2e-5 – 5e-5 ← low! don't break pretrain\n  batch_size: 16 – 32\n  epochs: 3 – 5 ← retraining quickly\n  warmup: 6% steps\n  weight_decay: 0.01\n\nData:\n  100+ examples → works\n  1000+ examples → good\n  10k+ examples → excellent"
            }
          ]
        },
        {
          "number": "14",
          "title": "Fine-tune encoder vs Prompt LLM: when and what",
          "tag": "Fine-tuning",
          "description": "It is easier to launch LLM through a prompt, but it is expensive and slow at large volumes. Fine-tuned encoder is smaller, faster, cheaper - and often more accurate for a specific task. The choice depends on the volume of traffic, data availability and latency requirements.",
          "blocks": [
            {
              "kind": "table",
              "headers": [
                "",
                "Fine-tuned Encoder",
                "LLM (prompting)"
              ],
              "rows": [
                [
                  "Data",
                  "need 100–10k examples",
                  "0 (zero-shot)"
                ],
                [
                  "Latency",
                  "2–20ms",
                  "200–2000ms"
                ],
                [
                  "Cost",
                  "~$0.00001/request",
                  "~$0.001–0.01/request"
                ],
                [
                  "Accuracy on task",
                  "high (specialization)",
                  "average (overall)"
                ],
                [
                  "Flexibility",
                  "only one task",
                  "any task"
                ],
                [
                  "Support",
                  "need MLOps",
                  "API"
                ]
              ]
            }
          ]
        },
        {
          "number": "15",
          "title": "Knowledge Distillation: small = fast",
          "tag": "Fine-tuning",
          "description": "Distillation is training a small “student” model to copy the behavior of a large “teacher” model. DistilBERT has 66% of BERT parameters, is 60% faster and maintains 97% quality. Ideal for production with strict latency requirements.",
          "blocks": [
            {
              "kind": "code",
              "label": "mechanics of distillation",
              "language": "text",
              "value": "Teacher: BERT-large (340M), accurate\nStudent: DistilBERT (66M), fast\n\nStudent Loss = combination of three goals:\n① Distillation loss:\n  KL(softmax(logits_student / T) ‖ softmax(logits_teacher / T))\n  T=temperature→ soft probabilities = “dark knowledge”\n\n② Cosine embedding loss:\n  hidden states of student ≈ teacher\n\n③ MLM loss:\n  standard training on masked tokens\n\nPopular distilled models:\n  DistilBERT: 66M, 60% faster, 97% BERT quality\n  TinyBERT: 15M, 9.4× faster, 96% quality\n  MobileBERT: 25M, edge deployment"
            }
          ]
        }
      ]
    },
    {
      "number": "05",
      "title": "Encoder-Decoder: T5, BART and Seq2Seq",
      "intro": "",
      "cards": [
        {
          "number": "16",
          "title": "Seq2Seq: encoder understands, decoder creates",
          "tag": "Encoder-Decoder",
          "description": "Encoder-decoder is an original transformer architecture from Attention Is All You Need. Encoder reads the entire input (bidirectional), builds its representation; decoder generates output using cross-attention to encoder representations. Ideal for \"convert A to B\" tasks.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "Architecture (example: translation):\n\n  Entry: \"The cat sat on the mat\"\n              ↓\n  ┌─────────────────── ────────────────────┐\n  │ ENCODER (bidirectional self-attention)│\n  │ h₁ h₂ h₃ h₄ h₅ h₆ │\n  │ The cat sat on the mat │\n  └─────────────────── ┬───────────────────┘\n                      │ cross-attention\n                      ↓ (decoder looks at ALL h encoder)\n  ┌─────────────────── ────────────────────┐\n  │ DECODER (causal + cross-attention) │\n  │ [START] → Cat → sat → on → … │\n  │ │\n  │ At every step: │\n  │ 1. Causal self-attn (past tokens) │\n  │ 2. Cross-attention to encoder h₁..h₆ │\n  │ 3. FFN → logits → next token │\n  └─────────────────── ────────────────────┘\n\nWhy is it better than decoder-only for translation:\n  → encoder reads the entire input phrase ENTIRELY\n  → decoder knows when generating each word\n     ALL source text via cross-attention\n  → less “forgetting” of long inputs"
            },
            {
              "kind": "list",
              "label": "problems where seq2seq is better than decoder-only LLM",
              "items": [
                "machine translation (fixed languages)",
                "summation with length control",
                "grammatical correction",
                "extracting structure from text",
                "requires more memory than encoder-only"
              ]
            }
          ]
        },
        {
          "number": "17",
          "title": "T5: «Text-to-Text Transfer Transformer»",
          "tag": "Encoder-Decoder",
          "description": "T5 (2020, Google) - unified frame: ALL tasks are formulated as “text → text”. Translation: “translate English to German: The cat...”. Summarize: “summarize: …”. This allows you to train one model on everything at once.",
          "blocks": [
            {
              "kind": "code",
              "label": "text-to-text frame",
              "language": "text",
              "value": "# All tasks are in the same format:\n\"translate English to Russian: Hello world\"\n  → \"Hello world\"\n\n\"summarize: [long text...]\"\n  → \"Summary: ...\"\n\n\"sst2 sentence: I love this movie\"\n  → \"positive\"\n\n\"cola sentence: The cat sat\"\n  → \"acceptable\"\n\nFLAN-T5 (2022):\n  → T5 + instruction tuning on 1000+ tasks\n  → follows instructions like LLM, but less\n  → flan-t5-large (780M) vs GPT-3 (175B)\n  → wins on structured generation"
            }
          ]
        },
        {
          "number": "18",
          "title": "BART: denoising pretraining",
          "tag": "Encoder-Decoder",
          "description": "BART (2020, Meta) - encoder-decoder with pre-training through denoising: the input is deliberately “spoilt” (fragments are removed, sentences are rearranged, spans are masked), the decoder learns to restore the original. Particularly good for summarization.",
          "blocks": [
            {
              "kind": "code",
              "label": "denoising tasks during training",
              "language": "sql",
              "value": "Original: \"The cat sat on the warm windowsill and watched\"\n\nToken masking: \"The cat [M] was on the [M] windowsill watching\"\nToken deletion: \"The cat was watching on the warm windowsill\"\nText infilling: \"The cat [M M M] was looking\" ← one [M] = many words\nSentence permut: \"I looked. The cat was sitting on the windowsill and\"\nDocument rotation: start from the middle\n\nWhy does this help summarization:\n  → infilling = generating a missing fragment\n  → summarization = generation of a “compressed” fragment\n  → structurally similar tasks\n\nmBART: multilingual BART, 25 languages, including ru"
            }
          ]
        }
      ]
    },
    {
      "number": "06",
      "title": "When a small model beats GPT-4",
      "intro": "",
      "cards": [
        {
          "number": "19",
          "title": "Triangle: Latency · Cost · Accuracy",
          "tag": "Comparison",
          "description": "Choosing a model is always a compromise of three parameters. GPT-4 is the most powerful in terms of accuracy, but the most expensive and slow. Fine-tuned RoBERTa is the fastest and cheapest. The key question is: How complex is your specific problem?",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "Real numbers (2025, typical production):\n\nModel Latency Cost/1M Accuracy Parameters\n───────────────────────────── ──────────────────────────────\nfine-tuned roberta 3ms $0.01 96% 125M\nfine-tuned deberta 8ms $0.02 97.5% 184M\nclaude-haiku-3-5 300ms $0.80 91% ~20B?\ngpt-4o-mini 400ms $0.60 93% ~20B?\nclaude-sonnet-4-6 1200ms $15.00 99% ~200B?\ngpt-4o 1500ms $10.00 98% ~200B?\n───────────────────────────── ──────────────────────────────\n* accuracy for the intent classification task with 5000 examples\n\nConclusion: fine-tuned RoBERTa is 100x cheaper and 400x faster\nGPT-4 gives only ~3% increase in accuracy.\nAt 1M requests/day: $10 vs $10,000 per day.\n\nWhen GPT-4 is needed:\n  → the task requires reasoning through several steps\n  → no marked data for fine-tuning\n  → the task changes every day\n  → need a detailed text answer"
            }
          ]
        },
        {
          "number": "20",
          "title": "Problems where encoder-only reliably wins",
          "tag": "Comparison",
          "description": "There are categories of tasks where specialized encoder models consistently outperform LLM in all three parameters - not only in speed and price, but also in accuracy. These are tasks with a clear structure and a sufficient amount of training data.",
          "blocks": [
            {
              "kind": "code",
              "label": "tasks with victory encoder",
              "language": "text",
              "value": "① Classification of intents (IntentRouter):\n  fine-tuned RoBERTa: 96.5% F1, 3ms\n  claude-haiku prompt: 91.2% F1, 280ms\n  → 100x faster, more precisely, 80x cheaper\n\n② NER (Entity Extraction):\n  fine-tuned BERT/DeBERTa: F1=0.92\n  GPT-4 prompting: F1=0.87\n  → encoder is more accurate on clearly labeled data\n\n③ Reranking in RAG:\n  cross-encoder reranker: NDCG@10=0.74\n  embedding cosine: NDCG@10=0.61\n  → cross-encoder sees query+doc together\n\n④ Semantic similarity (deduplication):\n  bi-encoder (E5): 50k par/sec\n  LLM: 10 pairs/sec\n  → scale is not possible with LLM\n\n⑤ Content moderation:\n  fine-tuned toxicity classifier: 0.5ms/request\n  → should be cheaper than the LLM request itself"
            }
          ]
        },
        {
          "number": "21",
          "title": "Cascade pipeline: small + large",
          "tag": "Comparison",
          "description": "The best strategy is not to choose one model, but to build a cascade: fast specialized models filter and route, LLM is called only where the “heavy artillery” is needed. This is exactly how production systems with high traffic work.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "User request\n       ↓\n[RoBERTa IntentClassifier] 3ms, $0.00001\n  intent=\"price_query\" (conf=0.96) → PriceAgent\n  intent=\"complaint\" (conf=0.91) → ComplaintAgent\n  intent=\"unclear\" (conf=0.43) → ↓ (LLM needed)\n                                              ↓\n                             [Claude Haiku] 280ms, $0.001\n                              clarification?\n                              simple_task? → answer directly\n                              complex?    → ↓\n                                           [Claude Sonnet] 1200ms, $0.015\n\nTypical traffic statistics:\n  70% → RoBERTa copes ($0.00001/request)\n  25% → Haiku ($0.001/request)\n   5% → Sonnet ($0.015/request)\n  Average cost: ~$0.002 (vs $0.015 if everything is on Sonnet)"
            }
          ]
        }
      ]
    },
    {
      "number": "07",
      "title": "Practical architecture selection map",
      "intro": "",
      "cards": [
        {
          "number": "22",
          "title": "Decision tree: task → architecture → model",
          "tag": "Selection card",
          "description": "Structured choice. Start with the type of task, take into account the availability of data and latency requirements - get a specific model recommendation.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "Need to GENERATE text?\n  No → continue below\n  Yes → fixed structure A→B? (translation, summarization)\n          Yes → Encoder-Decoder: T5, FLAN-T5, BART, mBART\n          No → free generation, reasoning, dialogue\n               → Decoder-only LLM: Claude, GPT-4o, Llama-3\n\n────────────────────────── ───────────────────────────\n\nTask type (without generation):\n\n  Text classification (intent, sentiment, topic, toxicity)\n    Are there 500+ examples?\n      Yes → fine-tuned RoBERTa / DeBERTa / ModernBERT\n      No → zero-shot Claude Haiku / GPT-4o-mini\n\n  Entity extraction (NER, spans)\n    Are there 200+ marked up offers?\n      Yes → fine-tuned BERT / RoBERTa (token classification)\n      No -> LLM with JSON output\n\n  Semantic embeddings (search, clustering, deduplication)\n    Do you need Multilingual?\n      Yes → multilingual-e5, paraphrase-multilingual-mpnet\n      No → E5-large-v2, BGE-large-en, nomic-embed-text\n\n  Reranking (clarification of top-K RAG results)\n    → cross-encoder: bge-reranker-v2, ms-marco-MiniLM\n\n  Sentence similarity/Textual Entailment\n    → fine-tuned DeBERTa-NLI, SBERT\n\n  Multilingual classification\n    → fine-tuned XLM-RoBERTa"
            }
          ]
        },
        {
          "number": "23",
          "title": "Quick Reference: models and their niches",
          "tag": "Selection card",
          "description": "A cheat sheet for specific models for quick selection at the start of the project.",
          "blocks": [
            {
              "kind": "table",
              "headers": [
                "Model",
                "Size",
                "Best use"
              ],
              "rows": [
                [
                  "ModernBERT-base",
                  "149M",
                  "classification, NER, 2025"
                ],
                [
                  "DeBERTa-v3-base",
                  "184M",
                  "NLI, QA, best EN encoder"
                ],
                [
                  "RoBERTa-base",
                  "125M",
                  "classification, fast"
                ],
                [
                  "XLM-RoBERTa-base",
                  "278M",
                  "multilingual, including RU"
                ],
                [
                  "E5-large-v2",
                  "335M",
                  "embeddings for EN search"
                ],
                [
                  "multilingual-e5",
                  "278M",
                  "multilingual embeddings"
                ],
                [
                  "bge-reranker-v2-m3",
                  "568M",
                  "reranking RAG"
                ],
                [
                  "FLAN-T5-large",
                  "780M",
                  "structured generation"
                ],
                [
                  "DistilBERT",
                  "66M",
                  "edge/mobile deployment"
                ]
              ]
            },
            {
              "kind": "list",
              "label": "all available at",
              "items": [
                "huggingface.co/models",
                "pip install transformers",
                "sentence-transformers for embeddings"
              ]
            }
          ]
        }
      ]
    }
  ]
}