LLM Internals: From Token to Logit to Answer

# Training BPE dictionary
1. Start with the character alphabet (bytes/unicode)
2. Count the frequency of all pairs of neighboring tokens
3. Merge the most common pair → new token
4. Repeat until the desired vocab_size

# Inference
"Hello world" → [15496, 995] # GPT-2
"hello" → […, …, …] # Cyrillic alphabet often breaks down to a larger number of tokens

E ∈ ℝ^[vocab_size × d_model] # embedding matrix

token_id = 15496 # "Hello"
x = E[token_id] # → vector ℝ^d_model

# GPT-3: d_model=12288, vocab=50257
# Llama-3 70B: d_model=8192, vocab=128256

# Parameters only in E:
50257 × 768 = 38.6M # GPT-2 base

Sinusoidal (original Transformer):
  PE(pos,2i) = sin(pos / 10000^2i/d_model)
  PE(pos,2i+1) = cos(pos / 10000^2i/d_model)
  → fixed, not trained

Learned (GPT-2, BERT):
  pos_emb ∈ ℝ^[max_seq × d_model]
  → trainable, limited by max_seq

RoPE (Llama, Mistral, GPT-NeoX):
  → rotation of Q,K in space by an angle θ·pos
  → relative positions, extrapolation
  → the best standard for today

Prompt: "Hi, how are you?"

┌─ Tokenize ──────────────────────────┐
│ [3, 45821, 77, 1234, 5] │
└─────────────────────────────────────┘
           ↓ E[token_id]
┌─ Token Emb ─────────────────────────┐
│ [0.2,-0.1,…] [-0.5,0.3,…] … │
└─────────────────────────────────────┘
           ↓ + pos_emb
┌─ x₀ (input to layer 0) ────────────┐
│ shape: [seq_len × d_model] │
│ example: [5 × 4096] for Llama-3 8B │
└─────────────────────────────────────┘

For each token x_i, we calculate three vectors through linear projections:

  Q_i = x_i W_Q (Query - “what am I looking for?”)
  K_j = x_j · W_K (Key - “what do I offer?”)
  V_j = x_j · W_V (Value - “what do I return?”)

Similarity score (score):
  score(i,j) = Q_i · K_jᵀ / √d_k
               ↑ division stabilizes gradients

Normalization and weighted sum:
  α_ij = softmax( score(i, 1..n) ) # “attention weights” (sum = 1)
  out_i = Σⱼ α_ij · V_j # weighted sum of values

Matrix form (the entire batch at once):
  Attention(Q,K,V) = softmax( Q·Kᵀ / √d_k ) · V

d_model = 4096 # Llama-3 8B
num_heads = 32
d_k = d_v = 128 # d_model / num_heads

W_Q: [4096 × 128] per head
W_K: [4096 × 128] per head
W_V: [4096 × 128] per head
W_O: [4096 × 4096] # output proj (all heads)

# Parameters in one attention layer:
4 × 4096² = 67M parameters

for h in range(num_heads):
  head_h = Attention(
    Q = x W_Q_h,
    K = x W_K_h,
    V = x W_V_h
  ) # → [seq × d_k]

# Concatenation of heads:
concat = cat([head_0, …, head_h]) # [seq × d_model]
out = concat W_O # [seq × d_model]

Matrix score (up to softmax), seq_len=4:

      t0 t1 t2 t3
t0 [ 2.1 -∞ -∞ -∞ ]
t1 [ 1.5 3.2 -∞ -∞ ]
t2 [ 0.8 1.1 2.7 -∞ ]
t3 [ 1.2 0.5 1.9 2.3 ]

↑ lower triangle - the model sees
↑ upper triangle (−∞) - prohibited

Example: "the cat sat on the mat"
Head 3 - coreference:

      the cat sat on the mat
cat [.05 .70 .08 .04 .05 .08]
sat [.04 .62 .20 .06 .04 .04]
mat [.08 .15 .12 .10 .10 .45]

High weight = "important token for this position"

x_in ───────────────────────────── ────────────────────────────┐
        ↓ │
  LayerNorm₁(x_in) # Pre-Norm (modern standard) │
        ↓ │
  Multi-Head Self-Attention │
        ↓ │ residual
x_attn = x_in + Attention_output ←──────────────────────────┘

x_attn ─────────────────────────── ───────────────────────────┐
        ↓ │
  LayerNorm₂(x_attn) │
        ↓ │ residual
  Feed-Forward Network (MLP) │
        ↓ │
x_out = x_attn + MLP_output ←─────────────────────────────-┘

# Original Transformer (ReLU):
FFN(x) = ReLU(x W₁ + b₁) W₂ + b₂
d_ff = 4 × d_model # expansion factor

# SwiGLU (Llama, PaLM - modern standard):
FFN(x) = SiLU(x W₁) ⊙ (x W₂) W₃
# three matrices: gate, up, down
d_ff = 8/3 × d_model ≈ 11k for d=4096

# Parameters in MLP (Llama-3 8B):
3 × 4096 × 11008 = 135M per block

LayerNorm(x) = γ · (x − μ) / (σ + ε) + β

  μ = mean(x) # by d_model
  σ = std(x) # by d_model
  ε = 1e-6 # numerical stability
  γ,β # trainees (scale/shift)

# RMSNorm (Llama, T5) - without subtracting the average:
RMSNorm(x) = γ x / RMS(x)
→ faster, quality is no worse

# Regular network:
x₃₂ = f₃₂(f₃₁(…f₁(x₀)…))
∂L/∂x₀ = ∏ᵢ ∂fᵢ/∂xᵢ ← product, → 0

# With residual:
x_{i+1} = x_i + F_i(x_i)
∂L/∂x₀ = Σᵢ 1 + ∂F_i/∂x_i ← sum, stable

# Each layer only learns the CORRECTION to x,
# not a full conversion → easier to optimize

x_last ∈ ℝ^d_model # last token vector

# LM Head = linear layer (usually without bias):
logits = x_last W_lm_headᵀ
# W_lm_head ∈ ℝ^[vocab_size × d_model]
# often = E (embedding matrix) - weight tying

logits ∈ ℝ^vocab_size # for example 128256

# Example logits (top 5):
"the" → 4.21
"a" → 3.87
"some" → 2.14
"this" → 1.95
"your" → 1.32 …

P(token_i) = exp(logit_i) / Σⱼ exp(logit_j)

# Example:
logits: [4.21, 3.87, 2.14, …]
  exp: [67.4, 47.9, 8.5, …]
 sum: 124.6 + …
probs: [0.32, 0.23, 0.04, …] ← sum = 1

# Numerical stability:
# subtract max(logits) before exp
P(i) = exp(logit_i - max) / Σ exp(logit_j - max)

P(i) = softmax( logits / T )

T = 0.0 → greedy decoding (argmax, determin.)
T = 0.3 → conservative, low variability
T = 0.7 → balance (OpenAI default)
T = 1.0 → “raw” model distribution
T = 1.5 → high randomness, creative chaos
T → ∞ → uniform (everything is equally likely)

top-k: leave only the top k tokens by probability
  k=50 → only 50 best options
  # problem: the tail can only be 3 tokens

top-p (nucleus): leave the minimum set
  with total probability ≥ p
  p=0.9 → take tokens while Σprob < 0.9
  # adaptive to the model's "confidence"

min-p: cut off tokens below the threshold
  min_p * p_max (Mistral recommendation)
  # less aggressive than top-p

# Typical chain:
logits → /T → top-k → top-p → softmax → sample

context = tokenize("Hi! How are you?")

while not stop_condition:
  logits = model.forward(context) # whole pass
  logits = logits[-1] # only last pose
  probes = sample(logits, T, top_p)
  token = draw(probs)
  context.append(token) # context grows!

# stop: EOS token or max_new_tokens
# problem: O(n²) without KV-cache

Without KV-cache (step 5):
  x = [t₀,t₁,t₂,t₃,t₄]
  compute K,V for ALL 5 tokens ← redundant!

With KV-cache:
  cache = {K:[k₀,k₁,k₂,k₃], V:[v₀,v₁,v₂,v₃]}
  new_k₄, new_v₄ = compute(t₄ only)
  cache.append(k₄, v₄)
  attention(q₄, cache.K, cache.V) ← O(1) per step

Cost of KV-cache memory (Llama-3 8B, seq=4096):
  2 × num_layers × num_kv_heads × seq × d_head × 2bytes
  = 2 × 32 × 8 × 4096 × 128 × 2 = ~536 MB

float32 (fp32): 4 bytes/parameter #learn
bfloat16 (bf16): 2 bytes/parameter # inference standard
float16 (fp16): 2 bytes/parameter # legacy, overflow risk
int8 (Q8): 1 byte/parameter # easy quantization
int4 (Q4): 0.5 bytes/parameter # GGUF, bitsandbytes
int2/1: 0.25 bytes # BitNet (experimental)

# Llama-3 70B parameters = 70×10⁹
bf16: 140 GB Q4: 35 GB Q2: 17 GB

x₀ = token_emb(tokens) + pos_emb
for l in range(num_layers):
  x_l = transformer_block_l(x_{l-1})

logits = lm_head(x_last) # [seq × vocab]
probes = softmax(logits)

# Teacher forcing: compare with true tokens
loss = cross_entropy(probs[:-1], tokens[1:])
# predict token t+1 given t, t-1, …, t0

∂L/∂W = ∂L/∂ŷ · ∂ŷ/∂z · ∂z/∂W

# Simplified for layer l:
δ_l = δ_{l+1} · ∂f_l/∂x_l # layer gradient l
∂L/∂W_l = δ_l x_{l-1}ᵀ # gradient of weights

# For attention (simplified):
∂L/∂W_Q = ∂L/∂out · ∂out/∂α · ∂α/∂score · Kᵀ

# Optimizer (AdamW):
W ← W - lr m̂ / (√v̂ + ε) - λ W

g = ∂L/∂W # gradient
m = β₁ m + (1-β₁) g # 1st moment (momentum)
v = β₂·v + (1-β₂)·g² # 2nd moment (variance)
m̂ = m / (1-β₁ᵗ) # bias correction
v̂ = v / (1-β₂ᵗ)
W ← W (1-lr λ) - lr m̂/(√v̂+ε) # weight decay separately

# Typical LLM hyperparameters:
β₁=0.9, β₂=0.95, ε=1e-8, λ=0.1
lr: 3e-4 (pretrain) → 1e-5 (finetune)

┌──────────────────────── ─────────────────────────┐
│ 1. DATA: raw text → tokens → batches │
│ batch: [B × seq_len] token ids │
├──────────────────────── ─────────────────────────┤
│ 2. FORWARD: tokens → logits (entire transformer) │
├──────────────────────── ─────────────────────────┤
│ 3. LOSS: cross_entropy(logits[:-1], tokens[1:]) │
│ predict each next token = NLL minimization │
├──────────────────────── ─────────────────────────┤
│ 4. BACKWARD: loss.backward() → gradients │
├──────────────────────── ─────────────────────────┤
│ 5. STEP: optimizer.step() → update weights │
│ gradient clipping: max_norm = 1.0 │
└──────────────────────── ─────────────────────────┘
          ↑ repeat 10⁶+ times

INPUT: "What is a neural network?"

  ①Tokenize ────────────────────── ───────────────────────
     [1337, 45821, 1003, 77, 234, 9821] (6 tokens)

  ② Token Embeddings ──────────────────────────────────────
     E[[1337, 45821, …]] → shape: [6 × 4096]

  ③ + Positional Embeddings (RoPE) ───────────────────────
     x₀ = token_emb + pos_emb → [6 × 4096]

  ④×32 Transformer Blocks ──────────────────────────────
     ┌─ block_l ──────────────────── ────────────────────┐
     │ RMSNorm → Multi-Head Self-Attention (32 heads) │
     │ + residual │
     │ RMSNorm → SwiGLU FFN (4096→11008→4096) │
     │ + residual │
     └───────────────────────── ─────────────────────────┘
     output: x₃₂ → [6 × 4096]

  ⑤LM Head ─────────────────────── ────────────────────────
     x₃₂[-1] · W_lm_headᵀ → logits [128256]

  ⑥ Temperature + Top-p ───────────────────────────────────
     logits / 0.7 → top_p(0.9) → softmax → probs [128256]

  ⑦ Sample ──────────────────────── ────────────────────────
     token = draw(probs) → "neural" (id: 77341)

  ⑧ Append & Repeat ──────────────────────────────────────
     context += [77341] # KV-cache updated, next step