{ "type": "new-runtime-knowledge-guide", "version": 1, "generated_at": "2026-07-23", "volume": 5, "slug": "llm-internals-reference", "title": "LLM Internals: From Token to Logit to Answer", "description": "The full path from the input request to the next token: tokenization, embeddings, positional encoding, self-attention, MLP, normalization, KV-cache, logits, softmax and sampling parameters - parsed atomically.", "track": { "slug": "llm-foundations", "title": "LLM Foundations", "route": "https://newruntime.com/learn/tracks/llm-foundations/", "position": 3 }, "counts": { "sections": 7, "cards": 28 }, "routes": { "html": "https://newruntime.com/learn/llm-internals-reference/", "json": "https://newruntime.com/learn/llm-internals-reference.json" }, "sections": [ { "number": "01", "title": "Input Pipeline", "intro": "", "cards": [ { "number": "01", "title": "Tokenization", "tag": "Input", "description": "The text cannot be fed directly to the neural network; it must be divided into integer indexes. A token is not always a word: GPT-4 uses ~100k tokens, where a token can be a syllable, a suffix, or a single character.", "blocks": [ { "kind": "code", "label": "how it works BPE (Byte-Pair Encoding)", "language": "text", "value": "# Training BPE dictionary\n1. Start with the character alphabet (bytes/unicode)\n2. Count the frequency of all pairs of neighboring tokens\n3. Merge the most common pair → new token\n4. Repeat until the desired vocab_size\n\n# Inference\n\"Hello world\" → [15496, 995] # GPT-2\n\"hello\" → […, …, …] # Cyrillic alphabet often breaks down to a larger number of tokens" }, { "kind": "list", "label": "facts", "items": [ "GPT-4: ~100k tokens", "Llama3: tiktoken BPE", "1 token ≈ 0.75 words (EN)", "Cyrillic: 1 word = 3–5 tokens" ] } ] }, { "number": "02", "title": "Token Embeddings", "tag": "Input", "description": "Each index token turns into a vector of dense numbers through a lookup table (weight matrix). This is the first learning layer - the model itself learns what each token means in the space of meanings.", "blocks": [ { "kind": "code", "label": "mechanics", "language": "text", "value": "E ∈ ℝ^[vocab_size × d_model] # embedding matrix\n\ntoken_id = 15496 # \"Hello\"\nx = E[token_id] # → vector ℝ^d_model\n\n# GPT-3: d_model=12288, vocab=50257\n# Llama-3 70B: d_model=8192, vocab=128256\n\n# Parameters only in E:\n50257 × 768 = 38.6M # GPT-2 base" }, { "kind": "list", "label": "properties", "items": [ "king − man + woman ≈ queen", "dot product = semantic proximity", "the same E is often used on the output (weight tying)" ] } ] }, { "number": "03", "title": "Positional Embeddings", "tag": "Input", "description": "The Attention mechanism itself is invariant to the order of tokens - it “sees” a set of vectors without the concept of “before/after”. Positional embeddings add position information directly to the token vector.", "blocks": [ { "kind": "code", "label": "three approaches", "language": "text", "value": "Sinusoidal (original Transformer):\n PE(pos,2i) = sin(pos / 10000^2i/d_model)\n PE(pos,2i+1) = cos(pos / 10000^2i/d_model)\n → fixed, not trained\n\nLearned (GPT-2, BERT):\n pos_emb ∈ ℝ^[max_seq × d_model]\n → trainable, limited by max_seq\n\nRoPE (Llama, Mistral, GPT-NeoX):\n → rotation of Q,K in space by an angle θ·pos\n → relative positions, extrapolation\n → the best standard for today" }, { "kind": "list", "label": "modern standard", "items": [ "RoPE — Llama, Mistral, Qwen", "ALiBi - length extrapolation", "Learned - limited by max_seq when learning", "Sinusoidal - deprecated" ] } ] }, { "number": "04", "title": "Input Representation", "tag": "Input", "description": "The final vector at the input of each transformer block is the sum of token embedding + positional embedding. It is this vector that “travels” through all layers, gradually enriching itself with context.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Prompt: \"Hi, how are you?\"\n\n┌─ Tokenize ──────────────────────────┐\n│ [3, 45821, 77, 1234, 5] │\n└─────────────────────────────────────┘\n ↓ E[token_id]\n┌─ Token Emb ─────────────────────────┐\n│ [0.2,-0.1,…] [-0.5,0.3,…] … │\n└─────────────────────────────────────┘\n ↓ + pos_emb\n┌─ x₀ (input to layer 0) ────────────┐\n│ shape: [seq_len × d_model] │\n│ example: [5 × 4096] for Llama-3 8B │\n└─────────────────────────────────────┘" } ] } ] }, { "number": "02", "title": "Attention Mechanism", "intro": "", "cards": [ { "number": "05", "title": "Self-Attention: mechanics", "tag": "Attention", "description": "Self-attention allows each token to “look” at all other tokens in the sequence and collect information in a weighted manner. This is the key mechanism by which the word \"key\" in the context of \"door key\" will be represented differently than in the context of \"forest key\".", "blocks": [ { "kind": "code", "label": "Diagram", "language": "tex", "value": "For each token x_i, we calculate three vectors through linear projections:\n\n Q_i = x_i W_Q (Query - “what am I looking for?”)\n K_j = x_j · W_K (Key - “what do I offer?”)\n V_j = x_j · W_V (Value - “what do I return?”)\n\nSimilarity score (score):\n score(i,j) = Q_i · K_jᵀ / √d_k\n ↑ division stabilizes gradients\n\nNormalization and weighted sum:\n α_ij = softmax( score(i, 1..n) ) # “attention weights” (sum = 1)\n out_i = Σⱼ α_ij · V_j # weighted sum of values\n\nMatrix form (the entire batch at once):\n Attention(Q,K,V) = softmax( Q·Kᵀ / √d_k ) · V" }, { "kind": "list", "label": "complexity", "items": [ "O(n²·d) in memory and time from context length", "Flash Attention - tile-based, O(n) memory", "d_k = d_model / num_heads" ] } ] }, { "number": "06", "title": "Q, K, V Projections", "tag": "Attention", "description": "Three weight matrices W_Q, W_K, W_V project the same input vector into three different “roles”. This allows the model to simultaneously decide “what I want to ask,” “what I have,” and “what to return in response.”", "blocks": [ { "kind": "code", "label": "dimensions", "language": "text", "value": "d_model = 4096 # Llama-3 8B\nnum_heads = 32\nd_k = d_v = 128 # d_model / num_heads\n\nW_Q: [4096 × 128] per head\nW_K: [4096 × 128] per head\nW_V: [4096 × 128] per head\nW_O: [4096 × 4096] # output proj (all heads)\n\n# Parameters in one attention layer:\n4 × 4096² = 67M parameters" } ] }, { "number": "07", "title": "Multi-Head Attention", "tag": "Attention", "description": "Instead of one attention, several (heads) are executed in parallel. Each head learns to pay attention to different aspects: one to syntactic connections, another to coreference, and a third to positional patterns.", "blocks": [ { "kind": "code", "label": "algorithm", "language": "python", "value": "for h in range(num_heads):\n head_h = Attention(\n Q = x W_Q_h,\n K = x W_K_h,\n V = x W_V_h\n ) # → [seq × d_k]\n\n# Concatenation of heads:\nconcat = cat([head_0, …, head_h]) # [seq × d_model]\nout = concat W_O # [seq × d_model]" }, { "kind": "list", "label": "variations", "items": [ "MHA is a classic, all Q/K/V are unique", "GQA — grouped query (Llama-3, Mistral)", "MQA - one K/V for all Q heads" ] } ] }, { "number": "08", "title": "Causal Mask (Decoder)", "tag": "Attention", "description": "In autoregressive (decoder-only) models, token i should not see tokens j > i. This is implemented by the mask: score(i,j) = −∞ for j > i, after softmax such positions receive a weight ≈ 0.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Matrix score (up to softmax), seq_len=4:\n\n t0 t1 t2 t3\nt0 [ 2.1 -∞ -∞ -∞ ]\nt1 [ 1.5 3.2 -∞ -∞ ]\nt2 [ 0.8 1.1 2.7 -∞ ]\nt3 [ 1.2 0.5 1.9 2.3 ]\n\n↑ lower triangle - the model sees\n↑ upper triangle (−∞) - prohibited" }, { "kind": "list", "label": "where there is no mask", "items": [ "BERT (encoder-only) - sees the entire context", "T5 (encoder-decoder) - encoder without mask" ] } ] }, { "number": "09", "title": "Attention Weights", "tag": "Attention", "description": "After softmax, the attention weights α_ij indicate how much token i “focuses” on token j. Row sum = 1. Different heads form different focus patterns - this can be visualized and interpreted.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Example: \"the cat sat on the mat\"\nHead 3 - coreference:\n\n the cat sat on the mat\ncat [.05 .70 .08 .04 .05 .08]\nsat [.04 .62 .20 .06 .04 .04]\nmat [.08 .15 .12 .10 .10 .45]\n\nHigh weight = \"important token for this position\"" }, { "kind": "list", "label": "interpretation tools", "items": [ "BertViz", "Transformer Lens", "not always = explanation, more of a heuristic" ] } ] } ] }, { "number": "03", "title": "Transformer Block", "intro": "", "cards": [ { "number": "10", "title": "Transformer Layer: Anatomy of a Block", "tag": "Block", "description": "Each transformer block (layer) is two subblocks with residual connections. First, multi-head attention “mixes” information between positions, then MLP “thinks” about each position independently. The 32-layer model runs the input tensor through this structure 32 times.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "x_in ───────────────────────────── ────────────────────────────┐\n ↓ │\n LayerNorm₁(x_in) # Pre-Norm (modern standard) │\n ↓ │\n Multi-Head Self-Attention │\n ↓ │ residual\nx_attn = x_in + Attention_output ←──────────────────────────┘\n\nx_attn ─────────────────────────── ───────────────────────────┐\n ↓ │\n LayerNorm₂(x_attn) │\n ↓ │ residual\n Feed-Forward Network (MLP) │\n ↓ │\nx_out = x_attn + MLP_output ←─────────────────────────────-┘" }, { "kind": "list", "label": "typical block parameters", "items": [ "Llama-3 8B: 32 blocks, d=4096", "Llama-3 70B: 80 blocks, d=8192", "each block: ~340M parameters (70B)" ] } ] }, { "number": "11", "title": "MLP / Feed-Forward Network", "tag": "Block", "description": "After attention, each token vector is processed independently by a two-layer neural network. MLP expands the dimension (×4 or ×8/3 for SwiGLU), applies nonlinearity, then contracts back. This is where the “actual memory” of the model is stored.", "blocks": [ { "kind": "code", "label": "classic FFN vs SwiGLU", "language": "text", "value": "# Original Transformer (ReLU):\nFFN(x) = ReLU(x W₁ + b₁) W₂ + b₂\nd_ff = 4 × d_model # expansion factor\n\n# SwiGLU (Llama, PaLM - modern standard):\nFFN(x) = SiLU(x W₁) ⊙ (x W₂) W₃\n# three matrices: gate, up, down\nd_ff = 8/3 × d_model ≈ 11k for d=4096\n\n# Parameters in MLP (Llama-3 8B):\n3 × 4096 × 11008 = 135M per block" }, { "kind": "list", "label": "interpretation", "items": [ "~⅔ model parameters - in MLP", "attention — routing, MLP — storage", "MoE replaces one MLP with multiple sparse" ] } ] }, { "number": "12", "title": "Layer Normalization", "tag": "Block", "description": "After each subblock, activations are normalized - the mean is subtracted, divided by std, then scaled by the trainable γ and β. This stabilizes learning and speeds up convergence, especially in deep networks.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "LayerNorm(x) = γ · (x − μ) / (σ + ε) + β\n\n μ = mean(x) # by d_model\n σ = std(x) # by d_model\n ε = 1e-6 # numerical stability\n γ,β # trainees (scale/shift)\n\n# RMSNorm (Llama, T5) - without subtracting the average:\nRMSNorm(x) = γ x / RMS(x)\n→ faster, quality is no worse" }, { "kind": "list", "label": "normalization position", "items": [ "Pre-Norm (before attention) - more stable", "Post-Norm (original) - requires warmup", "RMSNorm — Llama 2/3, Mistral, Qwen" ] } ] }, { "number": "13", "title": "Residual Connections", "tag": "Block", "description": "Each subblock adds its value to the input (skip connection). This solves the problem of vanishing gradient in deep networks - the gradient can flow directly through the skip path. Without residuals, it is almost impossible to train 32+ layers.", "blocks": [ { "kind": "code", "label": "why does this work", "language": "tex", "value": "# Regular network:\nx₃₂ = f₃₂(f₃₁(…f₁(x₀)…))\n∂L/∂x₀ = ∏ᵢ ∂fᵢ/∂xᵢ ← product, → 0\n\n# With residual:\nx_{i+1} = x_i + F_i(x_i)\n∂L/∂x₀ = Σᵢ 1 + ∂F_i/∂x_i ← sum, stable\n\n# Each layer only learns the CORRECTION to x,\n# not a full conversion → easier to optimize" } ] }, { "number": "14", "title": "Encoder vs Decoder vs Seq2Seq", "tag": "Block", "description": "The original Transformer had an encoder + decoder. Modern LLMs are almost always decoder-only. Key difference: decoder uses a causal mask and generates tokens autoregressively; encoder sees the full context at once.", "blocks": [ { "kind": "table", "headers": [ "Type", "Attention", "Task", "Example" ], "rows": [ [ "Encoder-only", "Bidirectional (no mask)", "Classification, embeddings", "BERT, RoBERTa" ], [ "Decoder-only", "Causal (lower triangle)", "Text generation", "GPT, Llama, Mistral" ], [ "Encoder-Decoder", "Cross-attention encoder→decoder", "Translation, summarization", "T5, BART, mT5" ] ] } ] } ] }, { "number": "04", "title": "Output & Sampling", "intro": "", "cards": [ { "number": "15", "title": "LM Head & Logits", "tag": "Output", "description": "After the last transformer block, the vector of the last token is projected into dictionary space. The result is a vector of logits of length vocab_size: raw, unnormalized scores for each possible next token.", "blocks": [ { "kind": "code", "label": "mechanics", "language": "text", "value": "x_last ∈ ℝ^d_model # last token vector\n\n# LM Head = linear layer (usually without bias):\nlogits = x_last W_lm_headᵀ\n# W_lm_head ∈ ℝ^[vocab_size × d_model]\n# often = E (embedding matrix) - weight tying\n\nlogits ∈ ℝ^vocab_size # for example 128256\n\n# Example logits (top 5):\n\"the\" → 4.21\n\"a\" → 3.87\n\"some\" → 2.14\n\"this\" → 1.95\n\"your\" → 1.32 …" } ] }, { "number": "16", "title": "Softmax → Probabilities", "tag": "Output", "description": "Softmax turns logits into a probability distribution: it exponentially amplifies large values and normalizes the sum to 1. It is from this distribution that the next token is sampled.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "P(token_i) = exp(logit_i) / Σⱼ exp(logit_j)\n\n# Example:\nlogits: [4.21, 3.87, 2.14, …]\n exp: [67.4, 47.9, 8.5, …]\n sum: 124.6 + …\nprobs: [0.32, 0.23, 0.04, …] ← sum = 1\n\n# Numerical stability:\n# subtract max(logits) before exp\nP(i) = exp(logit_i - max) / Σ exp(logit_j - max)" } ] }, { "number": "17", "title": "Temperature", "tag": "Output", "description": "Temperature divides the logits before softmax, controlling the \"sharpness\" of the distribution. T < 1 - the model is more confident and deterministic. T > 1 - the distribution is smoothed, the output is more random and varied.", "blocks": [ { "kind": "code", "label": "effect", "language": "text", "value": "P(i) = softmax( logits / T )\n\nT = 0.0 → greedy decoding (argmax, determin.)\nT = 0.3 → conservative, low variability\nT = 0.7 → balance (OpenAI default)\nT = 1.0 → “raw” model distribution\nT = 1.5 → high randomness, creative chaos\nT → ∞ → uniform (everything is equally likely)" }, { "kind": "list", "label": "recommendations", "items": [ "code / factual: T = 0.0–0.3", "chat: T = 0.6–0.8", "creative: T = 0.9–1.2", "T > 1.5 - often incoherent text" ] } ] }, { "number": "18", "title": "Top-k / Top-p / Min-p", "tag": "Output", "description": "Filters before sampling: cut off the “tail” of the distribution with unlikely tokens. Apply sequentially after temperature, until the final sample.", "blocks": [ { "kind": "code", "label": "algorithms", "language": "tex", "value": "top-k: leave only the top k tokens by probability\n k=50 → only 50 best options\n # problem: the tail can only be 3 tokens\n\ntop-p (nucleus): leave the minimum set\n with total probability ≥ p\n p=0.9 → take tokens while Σprob < 0.9\n # adaptive to the model's \"confidence\"\n\nmin-p: cut off tokens below the threshold\n min_p * p_max (Mistral recommendation)\n # less aggressive than top-p\n\n# Typical chain:\nlogits → /T → top-k → top-p → softmax → sample" } ] }, { "number": "19", "title": "Decoding Strategies", "tag": "Output", "description": "How exactly do you select a token from the distribution? Three fundamentally different approaches with different trade-offs between quality, variety and speed.", "blocks": [ { "kind": "table", "headers": [ "Method", "Idea", "+", "−" ], "rows": [ [ "Greedy", "argmax(probs)", "quickly, determin.", "repetition, suboptimal" ], [ "Beam Search", "B parallel hypotheses", "more global greedy", "slow, generic output" ], [ "Sampling", "sample(probs)", "variety", "need T, top-p tuning" ], [ "Contrastive", "max(sim_score)", "anti-repetition", "slower sampling" ] ] } ] }, { "number": "20", "title": "Autoregressive Generation", "tag": "Output", "description": "LLM generates one token per step, then adds it to the context and repeats. This is called autoregressive decoding. Each new token is completely dependent on all previous ones - hence the name \"language model\".", "blocks": [ { "kind": "code", "label": "generation cycle", "language": "python", "value": "context = tokenize(\"Hi! How are you?\")\n\nwhile not stop_condition:\n logits = model.forward(context) # whole pass\n logits = logits[-1] # only last pose\n probes = sample(logits, T, top_p)\n token = draw(probs)\n context.append(token) # context grows!\n\n# stop: EOS token or max_new_tokens\n# problem: O(n²) without KV-cache" } ] } ] }, { "number": "05", "title": "System Mechanics", "intro": "", "cards": [ { "number": "21", "title": "KV-Cache: how it works", "tag": "System", "description": "With autoregressive generation without optimization, K and V would have to be recalculated for all previous tokens at each step. KV-Cache saves the already calculated K and V, adding only a new token - this is a tens of times faster.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Without KV-cache (step 5):\n x = [t₀,t₁,t₂,t₃,t₄]\n compute K,V for ALL 5 tokens ← redundant!\n\nWith KV-cache:\n cache = {K:[k₀,k₁,k₂,k₃], V:[v₀,v₁,v₂,v₃]}\n new_k₄, new_v₄ = compute(t₄ only)\n cache.append(k₄, v₄)\n attention(q₄, cache.K, cache.V) ← O(1) per step\n\nCost of KV-cache memory (Llama-3 8B, seq=4096):\n 2 × num_layers × num_kv_heads × seq × d_head × 2bytes\n = 2 × 32 × 8 × 4096 × 128 × 2 = ~536 MB" }, { "kind": "list", "label": "optimizations on top of KV-cache", "items": [ "GQA/MQA - less K/V goals → less cache", "PagedAttention (vLLM) - virtual memory for KV", "Sliding Window - Mistral, limits cache", "Prefill phase: all prompt tokens together", "Decode phase: one token at a time" ] } ] }, { "number": "22", "title": "Prefill vs Decode Phases", "tag": "System", "description": "LLM inference is divided into two fundamentally different stages according to the nature of the load. Prefill - compute-bound, decode - memory-bound. This affects optimization of batching and GPU utilization.", "blocks": [ { "kind": "table", "headers": [ "", "Prefill", "Decode" ], "rows": [ [ "What", "Prompt processing", "Token generation" ], [ "Input data", "N tokens in parallel", "1 token per step" ], [ "Bottleneck", "Compute (matmul)", "Memory (KV load)" ], [ "Metrica", "TTFT (time-to-first-token)", "TPS (tokens/sec)" ] ] } ] }, { "number": "23", "title": "Numerical Precision", "tag": "System", "description": "Model parameters are stored in various floating point formats. The choice of format directly affects memory, speed and accuracy. Quantization allows the 70B model to run on a consumer GPU.", "blocks": [ { "kind": "code", "label": "formats and memory", "language": "text", "value": "float32 (fp32): 4 bytes/parameter #learn\nbfloat16 (bf16): 2 bytes/parameter # inference standard\nfloat16 (fp16): 2 bytes/parameter # legacy, overflow risk\nint8 (Q8): 1 byte/parameter # easy quantization\nint4 (Q4): 0.5 bytes/parameter # GGUF, bitsandbytes\nint2/1: 0.25 bytes # BitNet (experimental)\n\n# Llama-3 70B parameters = 70×10⁹\nbf16: 140 GB Q4: 35 GB Q2: 17 GB" } ] } ] }, { "number": "06", "title": "Training & Backprop", "intro": "", "cards": [ { "number": "24", "title": "Forward Pass", "tag": "Training", "description": "Direct pass: input tokens go through all layers, the output is the predicted probabilities for the next token. The loss function compares them with true tokens - this is cross-entropy loss.", "blocks": [ { "kind": "code", "label": "algorithm", "language": "python", "value": "x₀ = token_emb(tokens) + pos_emb\nfor l in range(num_layers):\n x_l = transformer_block_l(x_{l-1})\n\nlogits = lm_head(x_last) # [seq × vocab]\nprobes = softmax(logits)\n\n# Teacher forcing: compare with true tokens\nloss = cross_entropy(probs[:-1], tokens[1:])\n# predict token t+1 given t, t-1, …, t0" } ] }, { "number": "25", "title": "Backpropagation", "tag": "Training", "description": "After calculating the loss, the gradients are propagated in the opposite direction through all layers (chain rule). Each parameter receives a signal on how to change its value to reduce loss.", "blocks": [ { "kind": "code", "label": "chain rule in action", "language": "tex", "value": "∂L/∂W = ∂L/∂ŷ · ∂ŷ/∂z · ∂z/∂W\n\n# Simplified for layer l:\nδ_l = δ_{l+1} · ∂f_l/∂x_l # layer gradient l\n∂L/∂W_l = δ_l x_{l-1}ᵀ # gradient of weights\n\n# For attention (simplified):\n∂L/∂W_Q = ∂L/∂out · ∂out/∂α · ∂α/∂score · Kᵀ\n\n# Optimizer (AdamW):\nW ← W - lr m̂ / (√v̂ + ε) - λ W" }, { "kind": "list", "label": "gradient problems", "items": [ "vanishing: → 0 (deep networks without residuals)", "exploding: → ∞ (gradient clipping)", "residuals + LayerNorm solve both" ] } ] }, { "number": "26", "title": "AdamW Optimizer", "tag": "Training", "description": "Adam tracks the first (moment) and second (variance) moments of the gradients for an adaptive learning rate for each parameter. AdamW adds weight decay as a separate member, and not through gradient - this is important for regularization.", "blocks": [ { "kind": "code", "label": "AdamW algorithm", "language": "tex", "value": "g = ∂L/∂W # gradient\nm = β₁ m + (1-β₁) g # 1st moment (momentum)\nv = β₂·v + (1-β₂)·g² # 2nd moment (variance)\nm̂ = m / (1-β₁ᵗ) # bias correction\nv̂ = v / (1-β₂ᵗ)\nW ← W (1-lr λ) - lr m̂/(√v̂+ε) # weight decay separately\n\n# Typical LLM hyperparameters:\nβ₁=0.9, β₂=0.95, ε=1e-8, λ=0.1\nlr: 3e-4 (pretrain) → 1e-5 (finetune)" } ] }, { "number": "27", "title": "Training Pipeline Overview", "tag": "Training", "description": "Full cycle of LLM training: from tokenizing the corpus to updating the scales. Each step critically affects the final quality of the model. Modern LLMs are trained on trillions of tokens.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "┌──────────────────────── ─────────────────────────┐\n│ 1. DATA: raw text → tokens → batches │\n│ batch: [B × seq_len] token ids │\n├──────────────────────── ─────────────────────────┤\n│ 2. FORWARD: tokens → logits (entire transformer) │\n├──────────────────────── ─────────────────────────┤\n│ 3. LOSS: cross_entropy(logits[:-1], tokens[1:]) │\n│ predict each next token = NLL minimization │\n├──────────────────────── ─────────────────────────┤\n│ 4. BACKWARD: loss.backward() → gradients │\n├──────────────────────── ─────────────────────────┤\n│ 5. STEP: optimizer.step() → update weights │\n│ gradient clipping: max_norm = 1.0 │\n└──────────────────────── ─────────────────────────┘\n ↑ repeat 10⁶+ times" } ] } ] }, { "number": "07", "title": "Big Picture: Token Path", "intro": "", "cards": [ { "number": "28", "title": "Full path: Request → Next Token", "tag": "System", "description": "A summary diagram of everything that happens from the moment the text is received until the next token is issued. When generating a response of 200 tokens, this path goes through 200 times (prefill 1 time, decode 199 times).", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "INPUT: \"What is a neural network?\"\n\n ①Tokenize ────────────────────── ───────────────────────\n [1337, 45821, 1003, 77, 234, 9821] (6 tokens)\n\n ② Token Embeddings ──────────────────────────────────────\n E[[1337, 45821, …]] → shape: [6 × 4096]\n\n ③ + Positional Embeddings (RoPE) ───────────────────────\n x₀ = token_emb + pos_emb → [6 × 4096]\n\n ④×32 Transformer Blocks ──────────────────────────────\n ┌─ block_l ──────────────────── ────────────────────┐\n │ RMSNorm → Multi-Head Self-Attention (32 heads) │\n │ + residual │\n │ RMSNorm → SwiGLU FFN (4096→11008→4096) │\n │ + residual │\n └───────────────────────── ─────────────────────────┘\n output: x₃₂ → [6 × 4096]\n\n ⑤LM Head ─────────────────────── ────────────────────────\n x₃₂[-1] · W_lm_headᵀ → logits [128256]\n\n ⑥ Temperature + Top-p ───────────────────────────────────\n logits / 0.7 → top_p(0.9) → softmax → probs [128256]\n\n ⑦ Sample ──────────────────────── ────────────────────────\n token = draw(probs) → \"neural\" (id: 77341)\n\n ⑧ Append & Repeat ──────────────────────────────────────\n context += [77341] # KV-cache updated, next step" } ] } ] } ] }