{ "type": "new-runtime-knowledge-guide", "version": 1, "generated_at": "2026-07-23", "volume": 14, "slug": "llm-internals-kvcache-generation", "title": "LLM Internals: KV Cache & Generation", "description": "An in-depth analysis of the mechanics of inference: how exactly KV-cache works and why O(n²) without it, prefill vs decode phases, GPU memory arithmetic, GQA/MQA/PagedAttention, prompt caching at the API level (Anthropic + OpenAI), sampling from logits to token, speculative decoding, and the main case - “mother washed the frame” through LLM step by step.", "track": { "slug": "llm-foundations", "title": "LLM Foundations", "route": "https://newruntime.com/learn/tracks/llm-foundations/", "position": 4 }, "counts": { "sections": 8, "cards": 24 }, "routes": { "html": "https://newruntime.com/learn/llm-internals-kvcache-generation/", "json": "https://newruntime.com/learn/llm-internals-kvcache-generation.json" }, "sections": [ { "number": "01", "title": "KV-Cache: mechanics and why without it O(n²)", "intro": "", "cards": [ { "number": "01", "title": "Why is autoregressive generation without KV-cache catastrophically slow?", "tag": "KV-Cache", "description": "Decoder models generate one token at a time. Without optimization, each step requires a complete recalculation of attention for ALL previous tokens. 200 response tokens = 200 × N² operations. KV-cache is a solution that turns this into linear cost.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "WITHOUT KV-CACHE: quadratic hell\n\nGenerating the t₅ token requires attention throughout the entire sequence:\n Q = W_Q · [t₀, t₁, t₂, t₃, t₄, t₅] ← 6 tokens\n K = W_K · [t₀, t₁, t₂, t₃, t₄, t₅] ← 6 tokens\n V = W_V · [t₀, t₁, t₂, t₃, t₄, t₅] ← 6 tokens\n ↑ We are counting everything!\n\nTotal for 200 tokens: 1+2+3+...+200 = 20100 matmul steps\nComplexity: O(n² × d_model) ← disaster on long prompts\n\n───────────────────────────────── ─────────────────────────────────\nWith KV-CACHE: linear decode\n\nKey observation: K and V for t₀..t₄ do not change!\n Why recount what has already been counted?\n\nkv_cache = {\n K: [k₀, k₁, k₂, k₃, k₄], ← calculated once, stored\n V: [v₀, v₁, v₂, v₃, v₄], ← calculated once, stored\n}\n\nGeneration t₅:\n q₅ = W_Q · t₅ ← new token only\n k₅ = W_K · t₅ ← new token only\n v₅ = W_V · t₅ ← new token only\n kv_cache.K.append(k₅) ← update the cache\n kv_cache.V.append(v₅)\n out = attention(q₅, kv_cache.K, kv_cache.V) ← O(seq) per step\n\nTotal for 200 tokens: 200 matmul steps ← O(n) at the decode phase\nPrefill (prompt processing) is still O(n²) - but only once" }, { "kind": "list", "label": "practical effect", "items": [ "10-100× decode phase acceleration", "price: GPU memory - cache grows with length", "cache is stored per-layer, per-head" ] } ] }, { "number": "02", "title": "What is stored in KV-cache: anatomy", "tag": "KV-Cache", "description": "KV-cache is not a “query cache”, it is a specific data structure: two tensors K and V for each layer, each head, each token. Q is NOT put into the cache - only Q of the new token is needed for the next step.", "blocks": [ { "kind": "code", "label": "cache structure", "language": "text", "value": "# Cache form for one model:\nkv_cache.shape = [\n 2, #K and V (two tensors)\n num_layers, #32 for Llama-3 8B\n num_kv_heads, #8 (GQA) or 32 (MHA)\n seq_len, # current length (growing!)\n d_head #128 = 4096 / 32\n]\n\n# Q is NOT cached because:\n# Q is only needed to calculate the current step\n# q_i = W_Q t_i - only for new token i\n# K and V are ALWAYS needed with each new token\n\n# At each decode step:\nkv_cache.K[layer][head].append(k_new)\nkv_cache.V[layer][head].append(v_new)\n# seq_len increases by 1 → cache grows" } ] }, { "number": "03", "title": "Causal Mask and KV-cache: why Query should not be stored", "tag": "KV-Cache", "description": "Causal mask in the decoder-only model provides a key property: when generating token t_i, attention weights to t_i do not affect t₀..t_{i-1} - they have already been generated. This makes caching K and V correct.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "WHY IS IT MATHEMATICALLY CORRECT TO CACH K,V?\n\nCausal attention matrix (lower triangle):\n\n t₀ t₁ t₂ t₃\n t₀ [α₀₀ 0 0 0 ]\n t₁ [α₁₀ α₁₁ 0 0 ]\n t₂ [α₂₀ α₂₁ α₂₂ 0 ]\n t₃ [α₃₀ α₃₁ α₃₂ α₃₃]\n\nWhen calculating t₃, k₀,k₁,k₂ and v₀,v₁,v₂ are used\nThese values will not change in subsequent steps:\n k₄ will not be added to k₀, k₀ will remain k₀\n\n→ It is safe to cache K,V: they are immutable after writing\n→ Only K and V of the new step are added to the end of the cache\n→ Q of the new step is calculated on the fly and is not needed later" } ] }, { "number": "04", "title": "KV-cache and context length: what breaks at 128K", "tag": "KV-Cache", "description": "As the context window grows, KV-cache grows linearly. At 128K tokens, the cache becomes huge - more than the weights of the model itself. This is what makes long-context inference so expensive.", "blocks": [ { "kind": "code", "label": "cache growth at different lengths", "language": "text", "value": "# KV-cache size formula (fp16):\nkv_bytes = 2 × L × H_kv × S × d_h × 2\n\n# Llama-3 8B: L=32, H_kv=8 (GQA), d_h=128\nS= 4,096: 2 × 32 × 8 × 4096 × 128 × 2 = 537 MB\nS= 32,768: 2 × 32 × 8 × 32768 × 128 × 2 = 4.3 GB\nS=128,000: 2 × 32 × 8 × 128,000 × 128 × 2= 16.8 GB\n ↑ MORE models (8GB)\n\n# GPT-4 levels (96 layers, 128 heads, d_h=128, MHA):\nS=128,000: 2 × 96 × 128 × 128,000 × 128 × 2= ~640 GB\n ← requires several A100 only for KV-cache!" }, { "kind": "list", "label": "why GQA saves the day", "items": [ "GQA: H_kv=8 instead of 32 → 4× smaller cache", "MQA: H_kv=1 → 32× smaller cache", "compromise: slightly worse quality vs MHA" ] } ] } ] }, { "number": "02", "title": "Prefill vs Decode: two different worlds", "intro": "", "cards": [ { "number": "05", "title": "Prefill and Decode: fundamentally different bottlenecks", "tag": "Phases", "description": "LLM inference - two stages with opposite load characteristics. Prefill - matmul-bound: processes all prompt tokens in parallel, GPU is busy. Decode - memory-bound: each step reads the entire KV-cache from memory, the GPU is almost idle. Understanding the difference means optimizing correctly.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "PREFILL PHASE\n\n Input: entire prompt [t₀, t₁, ..., t_N] at once\n What it does: simultaneous forward pass for N tokens\n builds KV-cache for all positions\n Bottleneck: COMPUTE (matmul) ← GPU is 100% busy\n Metric: TTFT (Time-To-First-Token)\n Complexity: O(N²) — quadratic of prompt length\n\n \"mom washed the frame\" (4 tokens) → prefill 1 forward pass\n long RAG prompt (2000 tokens) → 500000× more expensive\n\n───────────────────────────────── ─────────────────────────────────\nDECODE PHASE\n\n Input: ONE token per step (last generated)\n What it does: forward for 1 token + reads the entire KV-cache\n adds k_new, v_new to cache\n Bottleneck: MEMORY BANDWIDTH ← reading KV-cache from HBM\n Metric: TPS / TPOT (Tokens Per Second, Time Per Output Token)\n Complexity: O(S) per step, O(N×S) total\n\n GPU utilization during decode: 5–20% ← GPU is empty!\n All the time is busy reading KV-cache from memory\n\n───────────────────────────────── ─────────────────────────────────\nLATENCY METRICS\n\n TTFT = Time To First Token ← prefill latency\n TPOT = Time Per Output Token ← decode speed\n TPS = Tokens Per Second ← 1/TPOT\n E2E = TTFT + N_output × TPOT ← total latency\n\n Streaming UX: TTFT is more important than E2E\n Batch offline: more important than TPS" }, { "kind": "list", "label": "Bottleneck-type optimizations", "items": [ "TTFT → shorten prompt / prompt caching", "TPS → reduce KV-cache (GQA/MQA)", "continuous batching → decode GPU utilization increases" ] } ] }, { "number": "06", "title": "Chunked Prefill: don't freeze decode", "tag": "Phases", "description": "Problem: a long prefill (2K+ tokens) blocks the GPU and delays TTFT for other users in the batch. Chunked prefill splits the prefill into parts, interspersed with decode - improves the uniformity of latency.", "blocks": [ { "kind": "code", "label": "mechanics", "language": "text", "value": "# Without chunked prefill:\nUser A prefill: [────────────────── 2000 tokens ──]\nUser B decode: [waiting]\n ← B is blocked for the entire time prefill A\n\n# With chunked prefill (chunk=256):\nUser A chunk 1: [──256──]\nUser B token 1: [─1─]\nUser A chunk 2: [──256──]\nUser B token 2: [─1─]\n...\n← A finishes a little later, but B doesn't freeze\n← latency tail (p99) improves significantly\n\n# Implementation: vLLM 0.4+, TensorRT-LLM" } ] }, { "number": "07", "title": "Continuous Batching vs Static Batching", "tag": "Phases", "description": "Static batching: requests are grouped by N and wait for the entire batch to complete. The problem is that the longest request in the batch makes everyone wait. Continuous batching: requests enter and exit the batch dynamically at each decode step.", "blocks": [ { "kind": "table", "headers": [ "", "Static", "Continuous" ], "rows": [ [ "Batch formation", "times in N requests", "each decode step" ], [ "New request included", "waiting for the next batch", "right away" ], [ "GPU utilization", "low at different lengths", "high" ], [ "Throughput", "2–3× less", "maximum" ], [ "Implementation", "just", "difficult (KV scheduling)" ], [ "Who uses", "old systems", "vLLM, TGI, TRT-LLM" ] ] } ] } ] }, { "number": "03", "title": "Memory Math: GQA, MQA, PagedAttention", "intro": "", "cards": [ { "number": "08", "title": "MHA → GQA → MQA: evolution of heads to save cache", "tag": "Memory", "description": "Multi-Head Attention keeps separate K and V for each head - maximum quality, maximum KV-cache. GQA and MQA - progressive reduction in the number of K/V goals while maintaining the number of Q goals. Llama-3, Mistral, Gemma use GQA.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "MHA: Multi-Head Attention (GPT-3, original transformer)\n\n num_heads_Q = 32\n num_heads_KV = 32 ← K,V for each Q-head\n\n Q-heads: [Q₀ Q₁ Q₂ ... Q₃₁]\n K-heads: [K₀ K₁ K₂ ... K₃₁] ← 32 separate Ks\n V-heads: [V₀ V₁ V₂ ... V₃₁] ← 32 individual Vs\n\n KV-cache size: 2 × 32 × S × d_h × 2 bytes ← basic\n\n───────────────────────────────── ─────────────────────────────────\nGQA: Grouped Query Attention (Llama-3, Mistral, Gemma)\n\n num_heads_Q = 32 ← same number of Q-heads\n num_heads_KV = 8 ← a group of 4 Q-heads shares one KV-head\n\n Q-groups: [Q₀ Q₁ Q₂ Q₃] [Q₄ Q₅ Q₆ Q₇] ... [Q₂₈..Q₃₁]\n K-heads: [K₀] [K₁] ... [K₇]\n V-heads: [V₀] [V₁] ... [V₇]\n\n KV-cache size: 2 × 8 × S × d_h × 2 bytes ← 4× smaller!\n Quality drop: minimal (GQA ≈ MHA on most benchmarks)\n\n───────────────────────────────── ─────────────────────────────────\nMQA: Multi-Query Attention (PaLM, Falcon)\n\n num_heads_Q = 32 ← same number of Q-heads\n num_heads_KV = 1 ← ALL Q-heads share ONE K,V pair\n\n KV-cache size: 2 × 1 × S × d_h × 2 bytes ← 32× less!\n Quality drop: noticeable - GQA is usually a better compromise" }, { "kind": "list", "label": "in production", "items": [ "Llama-3 8B: GQA (32Q / 8KV) → 2024 standard", "Mistral 7B: GQA (32Q / 8KV)", "MQA → niche, GQA displaces" ] } ] }, { "number": "09", "title": "PagedAttention (vLLM): virtual memory for KV", "tag": "Memory", "description": "The problem with traditional KV-cache: it is allocated as a contiguous block under max_seq_len. With a response of 512 tokens, a prompt of 100 tokens - 80% of the memory is idle. PagedAttention stores KV in continuous pages of ~16 tokens, allocating on demand.", "blocks": [ { "kind": "code", "label": "mechanics pages", "language": "text", "value": "# Traditional KV-cache:\nkv[req_1] = allocate(max_seq_len=2048) ← reserve everything at once\nkv[req_2] = allocate(max_seq_len=2048) ← 2048 more, although 100 are needed\n← 90% of GPU memory is fragmented with empty blocks\n\n# PagedAttention:\npage_size = 16 # tokens per page\n\nreq_1 ← [page_4, page_7, page_12] ← not physically adjacent\nreq_2 ← [page_1, page_9]\nreq_3 ← [page_2, page_5, page_11] ← allocated as they grow\n\n# block_table stores the mapping of logical → physical pages\n# GPU core vLLM can perform attention on non-adjacent blocks\n\nResult:\n memory waste: 30–40% (naive) → < 4% (PagedAttention)\n throughput: +2–4× with the same GPU" }, { "kind": "list", "label": "bonus: sharing pages", "items": [ "common system prompt → same page for all requests", "copy-on-write with branching (beam search)", "vLLM, TGI, SGLang use PagedAttention" ] } ] }, { "number": "10", "title": "Sliding Window Attention: limit cache growth", "tag": "Memory", "description": "An alternative approach: do not store KV for all past tokens - only for the last W. Each token sees only a window from the W previous ones. Mistral 7B uses SWA with W=4096. The cache stops growing after W tokens.", "blocks": [ { "kind": "code", "label": "tradeoff", "language": "text", "value": "Full KV-cache:\n attention(q_i, K[0..i], V[0..i]) ← all tokens\n maximum quality\n memory: O(seq_len) → grows without limit\n\nSliding Window (W=4096):\n attention(q_i, K[i-W..i], V[i-W..i]) ← window only\n quality: good for local dependencies\n memory: O(W) → constant after W tokens!\n\nSWA problem:\n token outside window = \"forgotten\"\n distant dependencies (beginning → end of text) are lost\n\nHybrid: some layers full attention,\n part of the SWA (Mistral Mixtral) layers" }, { "kind": "list", "label": "applies", "items": [ "Mistral 7B: W=4096 + full for multiple layers", "bad for tasks with long dependencies", "Longformer: SWA + global tokens (hybrid)" ] } ] } ] }, { "number": "04", "title": "Prompt Caching: KV-cache at the API level", "intro": "", "cards": [ { "number": "11", "title": "Prompt Caching: reuse prefill between requests", "tag": "Prompt Cache", "description": "A regular KV-cache lives inside a single request. Prompt Caching is a KV-cache between requests at the API level. If many requests have the same prefix (system prompt + RAG documents), the API provider stores them in KV-cache and does not recalculate them. Savings: up to 90% prefill cost and 85% TTFT latency.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "WITHOUT PROMPT CACHING: every time again\n\nRequest 1: [system_prompt=2000 tok] + [query_1=20 tok]\n → prefill 2020 tokens completely $$$\n\nRequest 2: [system_prompt=2000 tok] + [query_2=15 tok]\n → prefill 2015 tokens completely $$$\n ← 2000 tok is being recalculated!\n\n───────────────────────────────── ─────────────────────────────────\nWITH PROMPT CACHING: KV-cache on the server\n\nRequest 1: [system_prompt=2000 tok] + [query_1=20 tok]\n → prefill 2020 tokens cache_write\n → KV-cache for system_prompt is saved on the server\n\nRequest 2: [system_prompt=2000 tok] + [query_2=15 tok]\n → cache_hit: 2000 tokens from cache 10% cost\n → prefill only 15 new tokens → TTFT is 10x faster\n\n───────────────────────────────── ─────────────────────────────────\nKEY RULE OF USE\n\n Static → start of prompt (cached)\n Dynamic → end of prompt (not cached)\n\n ✓ Correct:\n [system_prompt] ← static → cached\n [rag_documents] ← static for TTL period → cached\n ────────────────\n [user_query] ← changes every time\n\n ✗ Incorrect:\n [user_query] ← first\n [system_prompt] ← after → is not cached!" }, { "kind": "list", "label": "when caching is especially effective", "items": [ "long system prompt (>1024 tokens)", "RAG with fixed documents", "few-shot examples in the prompt", "unique documents in each request - the cache will not help" ] } ] }, { "number": "12", "title": "Anthropic: cache_control API", "tag": "Prompt Cache", "description": "Anthropic requires explicit marking via cache_control. You decide which blocks to cache. Minimum 1024 tokens per cached block. TTL - 5 minutes (extends with each cache hit).", "blocks": [ { "kind": "code", "label": "request structure", "language": "sql", "value": "messages = client.messages.create(\n model = \"cloud-sonnet-4-6\",\n system = [\n {\n \"type\": \"text\",\n \"text\": system_prompt, ← static prompt\n \"cache_control\": {\"type\": \"ephemeral\"}\n }\n ],\n messages = [\n {\"role\": \"user\", \"content\": [\n {\"type\": \"text\", \"text\": rag_context,\n \"cache_control\": {\"type\": \"ephemeral\"}},\n {\"type\": \"text\", \"text\": user_query} ← not cacheable\n ]}\n ]\n)\n\n# Check the result:\nusage = messages.usage\nusage.cache_read_input_tokens # how many from cache\nusage.cache_creation_input_tokens # how many are written to the cache\nusage.input_tokens # regular (not cached)" }, { "kind": "list", "label": "Pricing (Claude 3.5 Sonnet)", "items": [ "cache_write: 125% of input price", "cache_read: 10% of input price (90% discount)", "pays off for >2 requests with one prefix" ] } ] }, { "number": "13", "title": "OpenAI: automatic prefix caching", "tag": "Prompt Cache", "description": "OpenAI (gpt-4o, gpt-4o-mini from Oct 2024) enables caching automatically - no markup is needed. The cache is activated at a prefix of 1024 tokens and a granularity of 128 tokens. TTL - 5-10 minutes.", "blocks": [ { "kind": "code", "label": "differences from Anthropic", "language": "text", "value": "OpenAI - automatically:\n ✓ No cache_control markup needed\n ✓ The API itself determines matching prefixes\n ✓ 50% discount on cached tokens (input)\n ? There is no explicit control - you don’t know what is cached\n\n# Check in response:\nresponse.usage.prompt_tokens_details\n → {\"cached_tokens\": 1920, \"audio_tokens\": 0}\n\nAnthropic - explicit markup:\n ✓ Full control over which block to cache\n ✓ 90% discount (vs 50% for OpenAI) → more aggressive\n ! We need to rebuild the prompt structure\n ! Minimum 1024 tokens per block\n\nAdvice: both providers - install static first,\ndynamic (query) to the end. It works for both." }, { "kind": "list", "label": "practice", "items": [ "OpenAI - easier to get started, no code changes", "Anthropic - more savings with correct markings", "both: block order is critical - static first" ] } ] } ] }, { "number": "05", "title": "Sampling: from logits to final token", "intro": "", "cards": [ { "number": "14", "title": "Full pipeline sampling: logits → temperature → top-k → top-p → sample", "tag": "Sampling", "description": "After forward pass, the model returns a vector of logits: one number for each token in the dictionary (128K for GPT-4). From these raw scores, you need to select one token. The chain of operations is deterministic, except for the final sample.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "tex", "value": "STEP 1: RAW LOGITS (after LM Head)\n\n logits = W_lm_head hidden_state[-1]\n shape: [128 256] ← vocab_size Llama-3\n\n Top 5 example:\n \"cat\" : 8.41\n \"dog\" : 6.23\n \"mouse\" : 5.87\n \"bird\" : 4.12\n ...128251 other tokens: -12 to +3\n\n───────────────────────────────── ─────────────────────────────────\nSTEP 2: TEMPERATURE (scaling)\n\n logits = logits / T\n\n T=0.0: logits/→∞ → argmax (greedy, deterministic)\n T=0.7: “cat”: 12.01, “dog”: 8.90 ← the gap has increased\n T=1.0: \"cat\": 8.41, \"dog\": 6.23 ← raw logits\n T=1.5: “cat”: 5.61, “dog”: 4.15 ← the gap has shrunk, chaos\n\n───────────────────────────────── ─────────────────────────────────\nSTEP 3: TOP-K (tail trimming by number)\n\n k=50: keep only 50 tokens with the highest logit\n the rest 128206 → logit = −∞\n\n───────────────────────────────── ─────────────────────────────────\nSTEP 4: TOP-P / NUCLEUS SAMPLING (trimming by total probability)\n\n After softmax we accumulate probabilities:\n \"cat\": 0.62 → cumsum = 0.62\n \"dog\": 0.18 → cumsum = 0.80\n \"mouse\": 0.09 → cumsum = 0.89\n \"bird\": 0.04 → cumsum = 0.93 ← if p=0.9, cut off here\n ...the rest → −∞\n\n p=0.9: keep tokens until Σprob ≤ 0.9\n\n───────────────────────────────── ─────────────────────────────────\nSTEP 5: SOFTMAX + SAMPLE\n\n probes = softmax(filtered_logits)\n next_token = random.choice(tokens, p=probs)\n ← the only non-deterministic step!" }, { "kind": "list", "label": "typical settings", "items": [ "code/factual: T=0, greedy (reproducible)", "chat: T=0.7, top_p=0.9", "creative: T=1.0–1.2, top_p=0.95", "T>1.5 → incoherent text" ] } ] }, { "number": "15", "title": "Temperature: physical intuition", "tag": "Sampling", "description": "The name \"temperature\" comes from thermodynamics. At high temperatures, molecules move randomly (high entropy). When low, it is orderly (low entropy). Similarly, temperature controls the entropy of the tokens' probability distribution.", "blocks": [ { "kind": "code", "label": "distributional effect", "language": "text", "value": "# Initial logits: [8.41, 6.23, 5.87, 4.12]\n\nT=0.3 (cold, “confident”):\n scaled: [28.0, 20.8, 19.6, 13.7]\n softmax: [0.993, 0.004, 0.002, 0.001]\n ← almost always \"cat\"\n\nT=1.0 (neutral, “raw distribution”):\n scaled: [8.41, 6.23, 5.87, 4.12]\n softmax: [0.617, 0.083, 0.058, 0.011]\n ← sometimes other options\n\nT=2.0 (hot, “chaotic”):\n scaled: [4.21, 3.12, 2.93, 2.06]\n softmax: [0.342, 0.189, 0.158, 0.071]\n ← often unexpected tokens\n\n# T→0: argmax (deterministic)\n# T→∞: uniform (everything is equally likely)" } ] }, { "number": "16", "title": "Greedy vs Beam Search vs Sampling", "tag": "Sampling", "description": "Three fundamentally different strategies for choosing the next token. Greedy is fast, Beam is high-quality for NMT, Sampling is diverse for generative tasks.", "blocks": [ { "kind": "table", "headers": [ "Strategy", "Logic", "Plus", "Minus" ], "rows": [ [ "Greedy", "argmax at each step", "fast, deterministic", "local optima" ], [ "Beam (B=4)", "store B best ways", "globally better greedy", "4x more expensive, repeats" ], [ "Sampling", "random from distribution", "variety, creative", "unstable" ], [ "Top-p sampling", "nucleus + chance", "balance of quality and diff.", "no" ] ] }, { "kind": "list", "label": "when what", "items": [ "translation/summarization: Beam B=4", "chat/creative: top-p sampling", "structured output: greedy (T=0)" ] } ] }, { "number": "17", "title": "Repetition Penalty & Logit Biases", "tag": "Sampling", "description": "LLM without additional mer tend to repeat tokens - especially at high temperatures. Repetition penalty reduces the probability of tokens that have already been encountered in the context. Logit bias allows you to force specific tokens to be strengthened or banned.", "blocks": [ { "kind": "code", "label": "mechanics", "language": "python", "value": "Repetition penalty (θ = 1.3):\n if token_id in context:\n logit[token_id] = logit[token_id] / 1.3 ← if > 0\n logit[token_id] = logit[token_id] * 1.3 ← if < 0\n # 1.0 = no penalty, 1.3 = moderate, 2.0 = aggressive\n\nLogit bias:\n # force deny token\n logit_bias = {token_id_of_swear: -100} ← = -∞\n\n # force the token to be strengthened\n logit_bias = {token_id_of_yes: +10} ← ≈ 20000× more likely\n\n# OpenAI API: logit_bias parameter\n# Usage: guardrail for banned words\n# or force structured output" } ] } ] }, { "number": "06", "title": "Generation Optimizations: speculative, quantization", "intro": "", "cards": [ { "number": "18", "title": "Speculative Decoding: a small model generates, a large one verifies", "tag": "Optimization", "description": "The main bottleneck of the decode phase is memory-bound: the GPU is idle waiting for data. Speculative decoding uses this time: a small “draft” model generates K tokens, a large model verifies them in ONE forward pass. Accepted tokens are free. Acceleration 2–3×.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "MECHANICS\n\nDraft model (haiku/Llama-3-1B):\n Generates K=5 tokens quickly:\n [\"neural\", \"network\", \"it\", \"mathematical\", \"function\"]\n Cost: 5 × 1B-forward-pass ← cheap\n\nTarget model (cloud-sonnet / Llama-3-70B):\n One forward pass by context + 5 draft tokens:\n Receives 5 probability distributions simultaneously\n Cost: 1 × 70B-forward-pass ← but validates 5 tokens!\n\nVERIFICATION (rejection sampling):\n\n Draft: [\"neural\", \"network\", \"it\", \"mathematical\", \"function\"]\n Target: P(neural)=0.71 P(network|...)=0.83 P(this|...)=0.04\n\n Token 1 \"neural\": draft P=0.7, target P=0.71 → ACCEPT\n Token 2 \"network\": draft P=0.8, target P=0.83 → ACCEPT\n Token 3 \"this\": draft P=0.6, target P=0.04 → REJECT\n → regenerate from position 3, discard the rest\n\nRESULT\n\n Traditional decode: 5 steps × 1 forward = 5 × 70B passes\n Speculative decode: 5 draft + 1 verify = 5 × 1B + 1 × 70B\n ≈ 2–3× faster with high acceptance rate\n\n Acceptance rate ≈ 70–80% with draft~target in the family\n Self-speculation: the model itself generates a draft based on early layers" }, { "kind": "list", "label": "when to use", "items": [ "latency is critical, draft and target are one family", "self-hosting: vLLM supports speculative decoding", "low temperature → higher acceptance rate", "when T>1.2: draft is often rejected → no profit" ] } ] }, { "number": "19", "title": "Quantization: fewer bits - less memory", "tag": "Optimization", "description": "Model parameters are stored in fp16 (2 bytes) or fp32 (4 bytes). Quantization reduces to int8 (1 byte) or int4 (0.5 byte). Llama-3 70B in fp16 = 140GB, int4 = 35GB - fits on two A100 40GB.", "blocks": [ { "kind": "code", "label": "formats and tradeoff", "language": "text", "value": "# Size of Llama-3 70B in different formats:\nfp32: 280 GB ← unrealistic\nfp16: 140 GB ← training\nbf16: 140 GB ← training (better fp16 for large numbers)\nint8: 70 GB ← almost no quality loss (LLM.int8)\nint4: 35 GB ← GPTQ/AWQ: slight loss of quality\nint3: 27 GB ← noticeable loss\nint2: 18 GB ← significant loss\n\n# KV-cache can also be quantized:\nfp16 → int8: KV-cache 2x smaller\n ← saves GPU memory for larger batches\n ← slight drift on long sequences" }, { "kind": "list", "label": "practice", "items": [ "int4 (GPTQ/AWQ): deploy 70B on 2×A40", "int8 (bitsandbytes): safe for production", "int4 KV-cache: carefully eval before deployment" ] } ] }, { "number": "20", "title": "Flash Attention: O(n) memory instead of O(n²)", "tag": "Optimization", "description": "The standard implementation of attention materializes a matrix S = QKᵀ of size n×n in HBM - with n=8192 this is 512MB for only one matrix. Flash Attention calculates attention by tile, never materializing the full matrix.", "blocks": [ { "kind": "code", "label": "key idea", "language": "tex", "value": "# Standard attention (slow, lots of memory):\nS = Q @ K.T / sqrt(d_k) # [n × n] in HBM - expensive!\nA = softmax(S) # [n × n] in HBM\nO=A@V#[n×d] in HBM\nMemory: O(n²)\n\n# Flash Attention (Dao et al. 2022):\n# Tile algorithm: Q, K, V blocks are loaded into SRAM\n# (fast GPU memory), softmax is calculated incrementally\n#S never fully materializes\nMemory: O(n) ← linear! only SRAM tiles\n\nResult:\n Speed: 2–4x faster on A100\n Memory: 5–20× less\n Numerically identical to standard attention" }, { "kind": "list", "label": "applies", "items": [ "all modern libraries: vLLM, HuggingFace", "critical for long-context (>8K tokens)", "FlashAttention-3 (2024): optimized for H100" ] } ] } ] }, { "number": "07", "title": "Token Embeddings vs Text Embeddings: Don't be confused!", "intro": "", "cards": [ { "number": "21", "title": "Token Embeddings (within LLM) vs Text Embeddings (for RAG) - different things", "tag": "Embeddings", "description": "One of the most common conceptual failures: mixing token embeddings (a table inside GPT, transforms ID→vector) and text embeddings (output of an encoder model of type text-embedding-3, representing all text as one vector for search). These are fundamentally different objects with different properties.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "TOKEN EMBEDDINGS (within LLM)\n\n What: matrix E ∈ ℝ^[vocab_size × d_model]\n lookup table: token_id → vector\n\n Input: integer token-ID (15496)\n Output: row vector of matrix E, dim = d_model (4096)\n Meaning: initial representation of a token in model space\n changes (enriches) on each transformer block\n\n Example:\n \"mother\" → id=77341 → E[77341] → [0.12, -0.34, 0.89, ...] ∈ ℝ^4096\n\n Usage: only inside the model, first layer\n NOT SUITABLE for search: not trained for retrieval task,\n context-sensitive (contextualized through attention)\n\n───────────────────────────────── ─────────────────────────────────\nTEXT EMBEDDINGS (for RAG/search)\n\n What: output of a specialized encoder model\n entire text → one vector representation\n\n Input: text of arbitrary length (up to max_seq_len)\n Output: one vector, dim = 768–3072 (depending on model)\n Meaning: semantic representation of the ENTIRE text\n specially trained: close in meaning → close vectors\n\n Example:\n \"mom washed the frame\" → encoder → [0.41, 0.23, -0.18, ...] ∈ ℝ^768\n cosine(\"mom washed the frame\", \"mom washed the frame\") → 0.97 ← close!\n\n Usage: indexing + search in RAG\n Models: text-embedding-3-large, bge-m3, e5-large\n\n───────────────────────────────── ─────────────────────────────────\nKEY DIFFERENCE\n\n Token embedding: \"bank\" → always the same vector ← static\n Text embedding: \"bank on the river\" → one vector ← context included\n \"bank account\" → another vector\n LLM hidden state: \"bank\" → different vector in different contexts ← contextualized" }, { "kind": "list", "label": "selection rule", "items": [ "RAG/search → text-embedding-3 / bge-m3", "fine-tuning models → token embeddings (inside)", "never use token embeddings from LLM for retrieval" ] } ] }, { "number": "22", "title": "Niche case: hidden states LLM as embedding", "tag": "Embeddings", "description": "Sometimes LLM hidden states are used as text embeddings - through prompt engineering. It works (LLM-Embedder, E5-mistral), but is niche: expensive (high inference), slow, justified only when maximum semantics is needed.", "blocks": [ { "kind": "code", "label": "when it makes sense", "language": "text", "value": "# E5-mistral: prompt engineering for retrieval\nprompt = f\"Instruct: Given a query, find relevant documents.\\n\nQuery: {query}\"\nembedding = llm_model.last_hidden_state(prompt)[-1]\n← take the last hidden state (pool)\n\n# When justified:\n✓ the task requires deep understanding (code, science)\n✓ has a GPU, latency < 1s is acceptable\n✓ bi-encoder embedding is not enough\n\n# Why standard text-embedding is better in 99% of cases:\n✗ LLM inference: 100–500ms vs 5–20ms for bi-encoder\n✗ cost 10–50× higher\n✗ bge-m3 on MTEB is often no worse than LLM-embedded" } ] } ] }, { "number": "08", "title": "End-to-End: “mom washed the frame” via GPT", "intro": "", "cards": [ { "number": "23", "title": "Each step: from “continue the text: mom washed the frame” to the first response token", "tag": "End-to-End", "description": "End-to-end parsing of one inference call. Let's take Llama-3 8B - specific numbers, specific forms of tensors. Task: continue the sentence.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "tex", "value": "INPUT: \"Continue the text: mom washed the frame\"\n────────────────────────────── ───────────────────────────────\n\n① TOKENIZATION (tokenizer: tiktoken / sentencepiece)\n \"Continue\" → [3643, 70878] ← 2 tokens (Cyrillic is expensive)\n \"text\" → [50078]\n \"mom\" → [86392]\n \"soap\" → [42341, 261] ← the word broke\n \"frame\" → [100342]\n\n input_ids = [3643, 70878, 50078, 86392, 42341, 261, 100342]\n seq_len = 7\n\n────────────────────────────── ───────────────────────────────\n② TOKEN EMBEDDINGS (lookup in matrix E ∈ ℝ^[128256 × 4096])\n E[3643] → x₀ ∈ ℝ^4096 ← vector for \"Continue\"\n E[70878] → x₁ ∈ ℝ^4096\n ...\n E[100342]→ x₆ ∈ ℝ^4096\n shape: [7 × 4096]\n\n────────────────────────────── ───────────────────────────────\n③ POSITIONAL EMBEDDINGS (RoPE)\n # RoPE is not applied as addition, but through rotation Q,K\n # when calculating attention inside each block\n # position is encoded into sine/cosine frequencies\n position_ids = [0, 1, 2, 3, 4, 5, 6]\n\n────────────────────────────── ───────────────────────────────\n④ 32 TRANSFORMER BLOCKS (repeat 32 times)\n\n ─ Block 0: ──────────────────────────────────────\n RMSNorm(x) → normalization: x / rms(x) γ\n\n GQA (32Q/8KV heads):\n Q = x W_Q shape: [7 × 32 × 128]\n K = x · W_K shape: [7 × 8 × 128] ← GQA: only 8!\n V = x W_V shape: [7 × 8 × 128]\n\n # K and V are placed in kv_cache[block_0]:\n kv_cache[0].K = K ← 7 tokens × 8 heads × 128 = 7168 float16\n kv_cache[0].V = V\n\n # Causal attention (lower triangle only):\n scores = Q Kᵀ / √128 shape: [7 × 32 × 7]\n scores += causal_mask ← +0 or −∞\n α = softmax(scores)\n attn_out = α V shape: [7 × 32 × 128]\n attn_out = reshape + W_O → [7 × 4096]\n\n Residual: x = x + attn_out\n\n RMSNorm(x)\n\n SwiGLU FFN: 4096 → 14336 → 4096\n gate = x W_gate [7 × 14336]\n up = x W_up [7 × 14336]\n act = SiLU(gate) * up\n down = act W_down [7 × 4096]\n\n Residual: x = x + down\n ─ ... repeated 31 times ───────────────────────\n\n Block output: x₃₂ ∈ ℝ^[7 × 4096]\n\n────────────────────────────── ───────────────────────────────\n⑤ LM HEAD (projection onto dictionary)\n x₃₂[-1] ← take only the last position “frame”\n logits = x₃₂[-1] · W_lm_headᵀ shape: [128256]\n\n Top 5 logits:\n \"erased\": 9.21 ← likely successor!\n \"rinsed\": 7.84\n \"washed\": 6.91\n \"cleaned\": 5.43\n \"rubs\": 4.12\n\n────────────────────────────── ───────────────────────────────\n⑥ SAMPLING (T=0.7, top_p=0.9)\n logits / 0.7 → [13.16, 11.20, 9.87, 7.76, 5.89, ...]\n softmax → [0.71, 0.12, 0.05, 0.02, 0.01, ...]\n top_p=0.9 → leave Σprob ≤ 0.9 for now\n sample() → \"erased\" (id: 91234)\n\n────────────────────────────── ───────────────────────────────\n⑦ NEXT STEP (DECODE)\n input_ids = [...7 tokens..., 91234] ← add a new one\n kv_cache contains K,V for the first 7 positions\n Compute only one new token: q₇, k₇, v₇\n kv_cache.append(k₇, v₇)\n ← decode step: O(1) mathmul, O(seq) cache read" } ] }, { "number": "24", "title": "Performance Numbers: Llama-3 8B on A100 80GB", "tag": "End-to-End", "description": "Specific numbers help calibrate expectations and explain in interviews why this or that optimization is important. Latency and throughput depend on batch size, length and accuracy.", "blocks": [ { "kind": "code", "label": "indicative metrics", "language": "text", "value": "# Llama-3 8B, fp16, A100 80GB SXM, vLLM\n\nPrefill (compute-bound):\n 1024 prompt tokens: TTFT ≈ 30 ms\n 4096 prompt tokens: TTFT ≈ 120 ms\n 16K prompt token: TTFT ≈ 900 ms ← O(n²) hits\n\nDecode (memory-bound):\n batch=1: TPS ≈ 80 tok/s\n batch=8: TPS ≈ 220 tok/s ← batching helps\n batch=32: TPS ≈ 400 tok/s ← saturates HBM bandwidth\n\nMemory:\n Model weights: ~16 GB (fp16)\n KV-cache @ 4K × batch=8: ~4.3 GB\n Activations: ~1–2 GB\n Total A100 80GB: fits with a margin\n\nWith optimizations:\n int4 (AWQ): 4 GB model, batch×2\n Speculative: latency 2–3× lower\n Prompt cache: TTFT 5–10× lower on hit" }, { "kind": "list", "label": "GPU memory budget", "items": [ "A100 80GB: 70B fp16 + KV-cache (batch=4)", "A100 40GB: 8B fp16 + KV-cache (batch=16)", "4×A100: 70B + large batch or long context" ] } ] } ] } ] }