LLM Internals: From Token to Logit to Answer

The text cannot be fed directly to the neural network; it must be divided into integer indexes. A token is not always a word: GPT-4 uses ~100k tokens, where a token can be a syllable, a suffix, or a single character.

Each index token turns into a vector of dense numbers through a lookup table (weight matrix). This is the first learning layer - the model itself learns what each token means in the space of meanings.

The Attention mechanism itself is invariant to the order of tokens - it “sees” a set of vectors without the concept of “before/after”. Positional embeddings add position information directly to the token vector.

The final vector at the input of each transformer block is the sum of token embedding + positional embedding. It is this vector that “travels” through all layers, gradually enriching itself with context.

Self-attention allows each token to “look” at all other tokens in the sequence and collect information in a weighted manner. This is the key mechanism by which the word "key" in the context of "door key" will be represented differently than in the context of "forest key".

Three weight matrices W_Q, W_K, W_V project the same input vector into three different “roles”. This allows the model to simultaneously decide “what I want to ask,” “what I have,” and “what to return in response.”

Instead of one attention, several (heads) are executed in parallel. Each head learns to pay attention to different aspects: one to syntactic connections, another to coreference, and a third to positional patterns.

In autoregressive (decoder-only) models, token i should not see tokens j > i. This is implemented by the mask: score(i,j) = −∞ for j > i, after softmax such positions receive a weight ≈ 0.

After softmax, the attention weights α_ij indicate how much token i “focuses” on token j. Row sum = 1. Different heads form different focus patterns - this can be visualized and interpreted.

Each transformer block (layer) is two subblocks with residual connections. First, multi-head attention “mixes” information between positions, then MLP “thinks” about each position independently. The 32-layer model runs the input tensor through this structure 32 times.

After attention, each token vector is processed independently by a two-layer neural network. MLP expands the dimension (×4 or ×8/3 for SwiGLU), applies nonlinearity, then contracts back. This is where the “actual memory” of the model is stored.

After each subblock, activations are normalized - the mean is subtracted, divided by std, then scaled by the trainable γ and β. This stabilizes learning and speeds up convergence, especially in deep networks.

Each subblock adds its value to the input (skip connection). This solves the problem of vanishing gradient in deep networks - the gradient can flow directly through the skip path. Without residuals, it is almost impossible to train 32+ layers.

The original Transformer had an encoder + decoder. Modern LLMs are almost always decoder-only. Key difference: decoder uses a causal mask and generates tokens autoregressively; encoder sees the full context at once.

After the last transformer block, the vector of the last token is projected into dictionary space. The result is a vector of logits of length vocab_size: raw, unnormalized scores for each possible next token.

Softmax turns logits into a probability distribution: it exponentially amplifies large values and normalizes the sum to 1. It is from this distribution that the next token is sampled.

Temperature divides the logits before softmax, controlling the "sharpness" of the distribution. T < 1 - the model is more confident and deterministic. T > 1 - the distribution is smoothed, the output is more random and varied.

Filters before sampling: cut off the “tail” of the distribution with unlikely tokens. Apply sequentially after temperature, until the final sample.

How exactly do you select a token from the distribution? Three fundamentally different approaches with different trade-offs between quality, variety and speed.

LLM generates one token per step, then adds it to the context and repeats. This is called autoregressive decoding. Each new token is completely dependent on all previous ones - hence the name "language model".

With autoregressive generation without optimization, K and V would have to be recalculated for all previous tokens at each step. KV-Cache saves the already calculated K and V, adding only a new token - this is a tens of times faster.

LLM inference is divided into two fundamentally different stages according to the nature of the load. Prefill - compute-bound, decode - memory-bound. This affects optimization of batching and GPU utilization.

Model parameters are stored in various floating point formats. The choice of format directly affects memory, speed and accuracy. Quantization allows the 70B model to run on a consumer GPU.

Direct pass: input tokens go through all layers, the output is the predicted probabilities for the next token. The loss function compares them with true tokens - this is cross-entropy loss.

After calculating the loss, the gradients are propagated in the opposite direction through all layers (chain rule). Each parameter receives a signal on how to change its value to reduce loss.

Adam tracks the first (moment) and second (variance) moments of the gradients for an adaptive learning rate for each parameter. AdamW adds weight decay as a separate member, and not through gradient - this is important for regularization.

Full cycle of LLM training: from tokenizing the corpus to updating the weights. Each step critically affects the final quality of the model. Modern LLMs are trained on trillions of tokens.

A summary diagram of everything that happens from the moment the text is received until the next token is issued. When generating a response of 200 tokens, this path goes through 200 times (prefill 1 time, decode 199 times).