--- type: "new-runtime-knowledge-guide" stable_id: "knowledge_guide:llm-internals-reference" version: 1 generated_at: "2026-07-23" record_date: "2026-07-23" date_kind: "generated_at" slug: "llm-internals-reference" title: "LLM Internals: From Token to Logit to Answer" description: "The full path from the input request to the next token: tokenization, embeddings, positional encoding, self-attention, MLP, normalization, KV-cache, logits, softmax and sampling parameters - parsed atomically." retrieval_nugget: "The full path from the input request to the next token: tokenization, embeddings, positional encoding, self-attention, MLP, normalization, KV-cache, logits, softmax and sampling parameters - parsed atomically. The text cannot be fed directly to the neural network; it must be divided into integer indexes." track: "llm-foundations" volume: 5 --- # LLM Internals: From Token to Logit to Answer ## Retrieval answer The full path from the input request to the next token: tokenization, embeddings, positional encoding, self-attention, MLP, normalization, KV-cache, logits, softmax and sampling parameters - parsed atomically. The text cannot be fed directly to the neural network; it must be divided into integer indexes. ## 01. Input Pipeline ### 01.01 Tokenization The text cannot be fed directly to the neural network; it must be divided into integer indexes. A token is not always a word: GPT-4 uses ~100k tokens, where a token can be a syllable, a suffix, or a single character. #### how it works BPE (Byte-Pair Encoding) ```text # Training BPE dictionary 1. Start with the character alphabet (bytes/unicode) 2. Count the frequency of all pairs of neighboring tokens 3. Merge the most common pair → new token 4. Repeat until the desired vocab_size # Inference "Hello world" → [15496, 995] # GPT-2 "hello" → […, …, …] # Cyrillic alphabet often breaks down to a larger number of tokens ``` #### facts - GPT-4: ~100k tokens - Llama3: tiktoken BPE - 1 token ≈ 0.75 words (EN) - Cyrillic: 1 word = 3–5 tokens ### 01.02 Token Embeddings Each index token turns into a vector of dense numbers through a lookup table (weight matrix). This is the first learning layer - the model itself learns what each token means in the space of meanings. #### mechanics ```text E ∈ ℝ^[vocab_size × d_model] # embedding matrix token_id = 15496 # "Hello" x = E[token_id] # → vector ℝ^d_model # GPT-3: d_model=12288, vocab=50257 # Llama-3 70B: d_model=8192, vocab=128256 # Parameters only in E: 50257 × 768 = 38.6M # GPT-2 base ``` #### properties - king − man + woman ≈ queen - dot product = semantic proximity - the same E is often used on the output (weight tying) ### 01.03 Positional Embeddings The Attention mechanism itself is invariant to the order of tokens - it “sees” a set of vectors without the concept of “before/after”. Positional embeddings add position information directly to the token vector. #### three approaches ```text Sinusoidal (original Transformer): PE(pos,2i) = sin(pos / 10000^2i/d_model) PE(pos,2i+1) = cos(pos / 10000^2i/d_model) → fixed, not trained Learned (GPT-2, BERT): pos_emb ∈ ℝ^[max_seq × d_model] → trainable, limited by max_seq RoPE (Llama, Mistral, GPT-NeoX): → rotation of Q,K in space by an angle θ·pos → relative positions, extrapolation → the best standard for today ``` #### modern standard - RoPE — Llama, Mistral, Qwen - ALiBi - length extrapolation - Learned - limited by max_seq when learning - Sinusoidal - deprecated ### 01.04 Input Representation The final vector at the input of each transformer block is the sum of token embedding + positional embedding. It is this vector that “travels” through all layers, gradually enriching itself with context. #### Diagram ```text Prompt: "Hi, how are you?" ┌─ Tokenize ──────────────────────────┐ │ [3, 45821, 77, 1234, 5] │ └─────────────────────────────────────┘ ↓ E[token_id] ┌─ Token Emb ─────────────────────────┐ │ [0.2,-0.1,…] [-0.5,0.3,…] … │ └─────────────────────────────────────┘ ↓ + pos_emb ┌─ x₀ (input to layer 0) ────────────┐ │ shape: [seq_len × d_model] │ │ example: [5 × 4096] for Llama-3 8B │ └─────────────────────────────────────┘ ``` ## 02. Attention Mechanism ### 02.05 Self-Attention: mechanics Self-attention allows each token to “look” at all other tokens in the sequence and collect information in a weighted manner. This is the key mechanism by which the word "key" in the context of "door key" will be represented differently than in the context of "forest key". #### Diagram ```tex For each token x_i, we calculate three vectors through linear projections: Q_i = x_i W_Q (Query - “what am I looking for?”) K_j = x_j · W_K (Key - “what do I offer?”) V_j = x_j · W_V (Value - “what do I return?”) Similarity score (score): score(i,j) = Q_i · K_jᵀ / √d_k ↑ division stabilizes gradients Normalization and weighted sum: α_ij = softmax( score(i, 1..n) ) # “attention weights” (sum = 1) out_i = Σⱼ α_ij · V_j # weighted sum of values Matrix form (the entire batch at once): Attention(Q,K,V) = softmax( Q·Kᵀ / √d_k ) · V ``` #### complexity - O(n²·d) in memory and time from context length - Flash Attention - tile-based, O(n) memory - d_k = d_model / num_heads ### 02.06 Q, K, V Projections Three weight matrices W_Q, W_K, W_V project the same input vector into three different “roles”. This allows the model to simultaneously decide “what I want to ask,” “what I have,” and “what to return in response.” #### dimensions ```text d_model = 4096 # Llama-3 8B num_heads = 32 d_k = d_v = 128 # d_model / num_heads W_Q: [4096 × 128] per head W_K: [4096 × 128] per head W_V: [4096 × 128] per head W_O: [4096 × 4096] # output proj (all heads) # Parameters in one attention layer: 4 × 4096² = 67M parameters ``` ### 02.07 Multi-Head Attention Instead of one attention, several (heads) are executed in parallel. Each head learns to pay attention to different aspects: one to syntactic connections, another to coreference, and a third to positional patterns. #### algorithm ```python for h in range(num_heads): head_h = Attention( Q = x W_Q_h, K = x W_K_h, V = x W_V_h ) # → [seq × d_k] # Concatenation of heads: concat = cat([head_0, …, head_h]) # [seq × d_model] out = concat W_O # [seq × d_model] ``` #### variations - MHA is a classic, all Q/K/V are unique - GQA — grouped query (Llama-3, Mistral) - MQA - one K/V for all Q heads ### 02.08 Causal Mask (Decoder) In autoregressive (decoder-only) models, token i should not see tokens j > i. This is implemented by the mask: score(i,j) = −∞ for j > i, after softmax such positions receive a weight ≈ 0. #### Diagram ```text Matrix score (up to softmax), seq_len=4: t0 t1 t2 t3 t0 [ 2.1 -∞ -∞ -∞ ] t1 [ 1.5 3.2 -∞ -∞ ] t2 [ 0.8 1.1 2.7 -∞ ] t3 [ 1.2 0.5 1.9 2.3 ] ↑ lower triangle - the model sees ↑ upper triangle (−∞) - prohibited ``` #### where there is no mask - BERT (encoder-only) - sees the entire context - T5 (encoder-decoder) - encoder without mask ### 02.09 Attention Weights After softmax, the attention weights α_ij indicate how much token i “focuses” on token j. Row sum = 1. Different heads form different focus patterns - this can be visualized and interpreted. #### Diagram ```text Example: "the cat sat on the mat" Head 3 - coreference: the cat sat on the mat cat [.05 .70 .08 .04 .05 .08] sat [.04 .62 .20 .06 .04 .04] mat [.08 .15 .12 .10 .10 .45] High weight = "important token for this position" ``` #### interpretation tools - BertViz - Transformer Lens - not always = explanation, more of a heuristic ## 03. Transformer Block ### 03.10 Transformer Layer: Anatomy of a Block Each transformer block (layer) is two subblocks with residual connections. First, multi-head attention “mixes” information between positions, then MLP “thinks” about each position independently. The 32-layer model runs the input tensor through this structure 32 times. #### Diagram ```text x_in ───────────────────────────── ────────────────────────────┐ ↓ │ LayerNorm₁(x_in) # Pre-Norm (modern standard) │ ↓ │ Multi-Head Self-Attention │ ↓ │ residual x_attn = x_in + Attention_output ←──────────────────────────┘ x_attn ─────────────────────────── ───────────────────────────┐ ↓ │ LayerNorm₂(x_attn) │ ↓ │ residual Feed-Forward Network (MLP) │ ↓ │ x_out = x_attn + MLP_output ←─────────────────────────────-┘ ``` #### typical block parameters - Llama-3 8B: 32 blocks, d=4096 - Llama-3 70B: 80 blocks, d=8192 - each block: ~340M parameters (70B) ### 03.11 MLP / Feed-Forward Network After attention, each token vector is processed independently by a two-layer neural network. MLP expands the dimension (×4 or ×8/3 for SwiGLU), applies nonlinearity, then contracts back. This is where the “actual memory” of the model is stored. #### classic FFN vs SwiGLU ```text # Original Transformer (ReLU): FFN(x) = ReLU(x W₁ + b₁) W₂ + b₂ d_ff = 4 × d_model # expansion factor # SwiGLU (Llama, PaLM - modern standard): FFN(x) = SiLU(x W₁) ⊙ (x W₂) W₃ # three matrices: gate, up, down d_ff = 8/3 × d_model ≈ 11k for d=4096 # Parameters in MLP (Llama-3 8B): 3 × 4096 × 11008 = 135M per block ``` #### interpretation - ~⅔ model parameters - in MLP - attention — routing, MLP — storage - MoE replaces one MLP with multiple sparse ### 03.12 Layer Normalization After each subblock, activations are normalized - the mean is subtracted, divided by std, then scaled by the trainable γ and β. This stabilizes learning and speeds up convergence, especially in deep networks. #### formula ```tex LayerNorm(x) = γ · (x − μ) / (σ + ε) + β μ = mean(x) # by d_model σ = std(x) # by d_model ε = 1e-6 # numerical stability γ,β # trainees (scale/shift) # RMSNorm (Llama, T5) - without subtracting the average: RMSNorm(x) = γ x / RMS(x) → faster, quality is no worse ``` #### normalization position - Pre-Norm (before attention) - more stable - Post-Norm (original) - requires warmup - RMSNorm — Llama 2/3, Mistral, Qwen ### 03.13 Residual Connections Each subblock adds its value to the input (skip connection). This solves the problem of vanishing gradient in deep networks - the gradient can flow directly through the skip path. Without residuals, it is almost impossible to train 32+ layers. #### why does this work ```tex # Regular network: x₃₂ = f₃₂(f₃₁(…f₁(x₀)…)) ∂L/∂x₀ = ∏ᵢ ∂fᵢ/∂xᵢ ← product, → 0 # With residual: x_{i+1} = x_i + F_i(x_i) ∂L/∂x₀ = Σᵢ 1 + ∂F_i/∂x_i ← sum, stable # Each layer only learns the CORRECTION to x, # not a full conversion → easier to optimize ``` ### 03.14 Encoder vs Decoder vs Seq2Seq The original Transformer had an encoder + decoder. Modern LLMs are almost always decoder-only. Key difference: decoder uses a causal mask and generates tokens autoregressively; encoder sees the full context at once. | Type | Attention | Task | Example | | --- | --- | --- | --- | | Encoder-only | Bidirectional (no mask) | Classification, embeddings | BERT, RoBERTa | | Decoder-only | Causal (lower triangle) | Text generation | GPT, Llama, Mistral | | Encoder-Decoder | Cross-attention encoder→decoder | Translation, summarization | T5, BART, mT5 | ## 04. Output & Sampling ### 04.15 LM Head & Logits After the last transformer block, the vector of the last token is projected into dictionary space. The result is a vector of logits of length vocab_size: raw, unnormalized scores for each possible next token. #### mechanics ```text x_last ∈ ℝ^d_model # last token vector # LM Head = linear layer (usually without bias): logits = x_last W_lm_headᵀ # W_lm_head ∈ ℝ^[vocab_size × d_model] # often = E (embedding matrix) - weight tying logits ∈ ℝ^vocab_size # for example 128256 # Example logits (top 5): "the" → 4.21 "a" → 3.87 "some" → 2.14 "this" → 1.95 "your" → 1.32 … ``` ### 04.16 Softmax → Probabilities Softmax turns logits into a probability distribution: it exponentially amplifies large values and normalizes the sum to 1. It is from this distribution that the next token is sampled. #### formula ```tex P(token_i) = exp(logit_i) / Σⱼ exp(logit_j) # Example: logits: [4.21, 3.87, 2.14, …] exp: [67.4, 47.9, 8.5, …] sum: 124.6 + … probs: [0.32, 0.23, 0.04, …] ← sum = 1 # Numerical stability: # subtract max(logits) before exp P(i) = exp(logit_i - max) / Σ exp(logit_j - max) ``` ### 04.17 Temperature Temperature divides the logits before softmax, controlling the "sharpness" of the distribution. T < 1 - the model is more confident and deterministic. T > 1 - the distribution is smoothed, the output is more random and varied. #### effect ```text P(i) = softmax( logits / T ) T = 0.0 → greedy decoding (argmax, determin.) T = 0.3 → conservative, low variability T = 0.7 → balance (OpenAI default) T = 1.0 → “raw” model distribution T = 1.5 → high randomness, creative chaos T → ∞ → uniform (everything is equally likely) ``` #### recommendations - code / factual: T = 0.0–0.3 - chat: T = 0.6–0.8 - creative: T = 0.9–1.2 - T > 1.5 - often incoherent text ### 04.18 Top-k / Top-p / Min-p Filters before sampling: cut off the “tail” of the distribution with unlikely tokens. Apply sequentially after temperature, until the final sample. #### algorithms ```tex top-k: leave only the top k tokens by probability k=50 → only 50 best options # problem: the tail can only be 3 tokens top-p (nucleus): leave the minimum set with total probability ≥ p p=0.9 → take tokens while Σprob < 0.9 # adaptive to the model's "confidence" min-p: cut off tokens below the threshold min_p * p_max (Mistral recommendation) # less aggressive than top-p # Typical chain: logits → /T → top-k → top-p → softmax → sample ``` ### 04.19 Decoding Strategies How exactly do you select a token from the distribution? Three fundamentally different approaches with different trade-offs between quality, variety and speed. | Method | Idea | + | − | | --- | --- | --- | --- | | Greedy | argmax(probs) | quickly, determin. | repetition, suboptimal | | Beam Search | B parallel hypotheses | more global greedy | slow, generic output | | Sampling | sample(probs) | variety | need T, top-p tuning | | Contrastive | max(sim_score) | anti-repetition | slower sampling | ### 04.20 Autoregressive Generation LLM generates one token per step, then adds it to the context and repeats. This is called autoregressive decoding. Each new token is completely dependent on all previous ones - hence the name "language model". #### generation cycle ```python context = tokenize("Hi! How are you?") while not stop_condition: logits = model.forward(context) # whole pass logits = logits[-1] # only last pose probes = sample(logits, T, top_p) token = draw(probs) context.append(token) # context grows! # stop: EOS token or max_new_tokens # problem: O(n²) without KV-cache ``` ## 05. System Mechanics ### 05.21 KV-Cache: how it works With autoregressive generation without optimization, K and V would have to be recalculated for all previous tokens at each step. KV-Cache saves the already calculated K and V, adding only a new token - this is a tens of times faster. #### Diagram ```text Without KV-cache (step 5): x = [t₀,t₁,t₂,t₃,t₄] compute K,V for ALL 5 tokens ← redundant! With KV-cache: cache = {K:[k₀,k₁,k₂,k₃], V:[v₀,v₁,v₂,v₃]} new_k₄, new_v₄ = compute(t₄ only) cache.append(k₄, v₄) attention(q₄, cache.K, cache.V) ← O(1) per step Cost of KV-cache memory (Llama-3 8B, seq=4096): 2 × num_layers × num_kv_heads × seq × d_head × 2bytes = 2 × 32 × 8 × 4096 × 128 × 2 = ~536 MB ``` #### optimizations on top of KV-cache - GQA/MQA - less K/V goals → less cache - PagedAttention (vLLM) - virtual memory for KV - Sliding Window - Mistral, limits cache - Prefill phase: all prompt tokens together - Decode phase: one token at a time ### 05.22 Prefill vs Decode Phases LLM inference is divided into two fundamentally different stages according to the nature of the load. Prefill - compute-bound, decode - memory-bound. This affects optimization of batching and GPU utilization. | | Prefill | Decode | | --- | --- | --- | | What | Prompt processing | Token generation | | Input data | N tokens in parallel | 1 token per step | | Bottleneck | Compute (matmul) | Memory (KV load) | | Metrica | TTFT (time-to-first-token) | TPS (tokens/sec) | ### 05.23 Numerical Precision Model parameters are stored in various floating point formats. The choice of format directly affects memory, speed and accuracy. Quantization allows the 70B model to run on a consumer GPU. #### formats and memory ```text float32 (fp32): 4 bytes/parameter #learn bfloat16 (bf16): 2 bytes/parameter # inference standard float16 (fp16): 2 bytes/parameter # legacy, overflow risk int8 (Q8): 1 byte/parameter # easy quantization int4 (Q4): 0.5 bytes/parameter # GGUF, bitsandbytes int2/1: 0.25 bytes # BitNet (experimental) # Llama-3 70B parameters = 70×10⁹ bf16: 140 GB Q4: 35 GB Q2: 17 GB ``` ## 06. Training & Backprop ### 06.24 Forward Pass Direct pass: input tokens go through all layers, the output is the predicted probabilities for the next token. The loss function compares them with true tokens - this is cross-entropy loss. #### algorithm ```python x₀ = token_emb(tokens) + pos_emb for l in range(num_layers): x_l = transformer_block_l(x_{l-1}) logits = lm_head(x_last) # [seq × vocab] probes = softmax(logits) # Teacher forcing: compare with true tokens loss = cross_entropy(probs[:-1], tokens[1:]) # predict token t+1 given t, t-1, …, t0 ``` ### 06.25 Backpropagation After calculating the loss, the gradients are propagated in the opposite direction through all layers (chain rule). Each parameter receives a signal on how to change its value to reduce loss. #### chain rule in action ```tex ∂L/∂W = ∂L/∂ŷ · ∂ŷ/∂z · ∂z/∂W # Simplified for layer l: δ_l = δ_{l+1} · ∂f_l/∂x_l # layer gradient l ∂L/∂W_l = δ_l x_{l-1}ᵀ # gradient of weights # For attention (simplified): ∂L/∂W_Q = ∂L/∂out · ∂out/∂α · ∂α/∂score · Kᵀ # Optimizer (AdamW): W ← W - lr m̂ / (√v̂ + ε) - λ W ``` #### gradient problems - vanishing: → 0 (deep networks without residuals) - exploding: → ∞ (gradient clipping) - residuals + LayerNorm solve both ### 06.26 AdamW Optimizer Adam tracks the first (moment) and second (variance) moments of the gradients for an adaptive learning rate for each parameter. AdamW adds weight decay as a separate member, and not through gradient - this is important for regularization. #### AdamW algorithm ```tex g = ∂L/∂W # gradient m = β₁ m + (1-β₁) g # 1st moment (momentum) v = β₂·v + (1-β₂)·g² # 2nd moment (variance) m̂ = m / (1-β₁ᵗ) # bias correction v̂ = v / (1-β₂ᵗ) W ← W (1-lr λ) - lr m̂/(√v̂+ε) # weight decay separately # Typical LLM hyperparameters: β₁=0.9, β₂=0.95, ε=1e-8, λ=0.1 lr: 3e-4 (pretrain) → 1e-5 (finetune) ``` ### 06.27 Training Pipeline Overview Full cycle of LLM training: from tokenizing the corpus to updating the weights. Each step critically affects the final quality of the model. Modern LLMs are trained on trillions of tokens. #### Diagram ```text ┌──────────────────────── ─────────────────────────┐ │ 1. DATA: raw text → tokens → batches │ │ batch: [B × seq_len] token ids │ ├──────────────────────── ─────────────────────────┤ │ 2. FORWARD: tokens → logits (entire transformer) │ ├──────────────────────── ─────────────────────────┤ │ 3. LOSS: cross_entropy(logits[:-1], tokens[1:]) │ │ predict each next token = NLL minimization │ ├──────────────────────── ─────────────────────────┤ │ 4. BACKWARD: loss.backward() → gradients │ ├──────────────────────── ─────────────────────────┤ │ 5. STEP: optimizer.step() → update weights │ │ gradient clipping: max_norm = 1.0 │ └──────────────────────── ─────────────────────────┘ ↑ repeat 10⁶+ times ``` ## 07. Big Picture: Token Path ### 07.28 Full path: Request → Next Token A summary diagram of everything that happens from the moment the text is received until the next token is issued. When generating a response of 200 tokens, this path goes through 200 times (prefill 1 time, decode 199 times). #### Diagram ```text INPUT: "What is a neural network?" ①Tokenize ────────────────────── ─────────────────────── [1337, 45821, 1003, 77, 234, 9821] (6 tokens) ② Token Embeddings ────────────────────────────────────── E[[1337, 45821, …]] → shape: [6 × 4096] ③ + Positional Embeddings (RoPE) ─────────────────────── x₀ = token_emb + pos_emb → [6 × 4096] ④×32 Transformer Blocks ────────────────────────────── ┌─ block_l ──────────────────── ────────────────────┐ │ RMSNorm → Multi-Head Self-Attention (32 heads) │ │ + residual │ │ RMSNorm → SwiGLU FFN (4096→11008→4096) │ │ + residual │ └───────────────────────── ─────────────────────────┘ output: x₃₂ → [6 × 4096] ⑤LM Head ─────────────────────── ──────────────────────── x₃₂[-1] · W_lm_headᵀ → logits [128256] ⑥ Temperature + Top-p ─────────────────────────────────── logits / 0.7 → top_p(0.9) → softmax → probs [128256] ⑦ Sample ──────────────────────── ──────────────────────── token = draw(probs) → "neural" (id: 77341) ⑧ Append & Repeat ────────────────────────────────────── context += [77341] # KV-cache updated, next step ```