--- type: "new-runtime-knowledge-guide" stable_id: "knowledge_guide:neuron-to-llm-reference" version: 1 generated_at: "2026-07-23" record_date: "2026-07-23" date_kind: "generated_at" slug: "neuron-to-llm-reference" title: "From a Neuron to an LLM" description: "The route from linear regression to transformer: classical ML models and their limit, biological neuron and perceptron, deep networks and representations, recurrence and its problems, attention as a breakthrough, transformer as architecture and LLM as a result of scale." retrieval_nugget: "The route from linear regression to transformer: classical ML models and their limit, biological neuron and perceptron, deep networks and representations, recurrence and its problems, attention as a breakthrough, transformer as architecture and LLM as a result of scale. ML is not a “smart program”." track: "llm-foundations" volume: 7 --- # From a Neuron to an LLM ## Retrieval answer The route from linear regression to transformer: classical ML models and their limit, biological neuron and perceptron, deep networks and representations, recurrence and its problems, attention as a breakthrough, transformer as architecture and LLM as a result of scale. ML is not a “smart program”. ## 01. Classic ML: models before neural networks ### 01.01 The essence of machine learning ML is not a “smart program”. This is the selection of parameters of a mathematical function so that it correctly maps the input to the output. Difference with conventional programming: rules are not written manually, they are calculated from data. #### three ways to write a program ```sql Normal programming: rules = person writes by hand output = rules(input) → works as long as there are few rules and they are clear Machine learning: model = fit(inputs, outputs) ← from data! output = model(new_input) → rules are found by the algorithm itself Deep learning (ML subtype): model = very flexible function (neural network) → finds rules even where a person cannot formulate them ``` #### key insight - data + algorithm = model - model = function with trained parameters ### 01.02 Linear models and their ceiling Linear regression, logistic regression, SVM all construct a straight line (or hyperplane) separating classes. They work great on simple tasks, but the real world is nonlinear. How to distinguish a cat from a dog with a straight line by pixels? #### Diagram ```text Linear model: ŷ = w₁ x₁ + w₂ x₂ + … + wₙ xₙ + b ↑ each sign is multiplied by weight Feature space (2D): x₂ ▲ ○ ○ ○ ← class A │ ○ ○ ○ │ ╱ ○ │ ╱ ← straight boundary │ ╱ ● ● │╱ ● ● ← class B └──────────────▶ x₁ Problem: x₂ ▲ ○ ● ○ ← XOR: no straight line! │ ● ○ ● └──────────────▶ x₁ Linear model → 50% accuracy (random) ``` #### what did you do before - feature engineering: manually created x² and x₁ x₂ - requires expert knowledge of the task - does not scale to images/text ### 01.03 The Problem of Manual Features Classic models - SVM, Random Forest, Gradient Boosting - are excellent when the characteristics are well defined: price, area, age. But using raw data (pixels, symbols, sound), you first need to come up with signs. This is called feature engineering - and this is where they hit the ceiling. #### classic NLP pipeline to neural networks ```sql # Task: determine the sentiment of the review Hand signs (2000s): "good" → +1, "bad" → -1 number of exclamation marks → feature TF-IDF by dictionary → sparse vector 50k sentence length, capital letters... Problem 1: "not bad" = "good"? Problem 2: "Test!" — positive or sarcasm? Problem 3: new words, slang → not in the dictionary → a person cannot come up with ALL the necessary signs → the neural network finds them itself from the data ``` ## 02. Neuron and Network: where do the names come from? ### 02.04 Biological neuron → model Neural networks are so named because they are mathematically inspired by the neurons of the brain. A biological neuron collects signals from dendrites, sums them up, and if the sum exceeds a threshold, it “shoots” through the axon. It is this logic that is reproduced by an artificial neuron. #### Diagram ```typescript Biological neuron: dendrite₁ ──signal──┐ dendrite₂ ──signal──┤ dendrite₃ ──signal──┤──▶ [cell body] ──▶ axon ──▶ next neuron dendrite₄ ──signal──┘ sums if sum > threshold Artificial Neuron (Perceptron, 1958): x₁ ──w₁──┐ x₂ ──w₂──┤ x₃ ──w₃──┤──▶ Σ(wᵢ·xᵢ) + b ──▶ f(z) ──▶ output x₄ ──w₄──┘ weighted sum function activation ``` #### analogy - dendrite = input feature x - synapse strength = weight w - response threshold = activation function - the analogy is simplified - the brain is more complex ### 02.05 Neuron mathematics One neuron is a linear function plus nonlinearity. The linear part weighs the inputs; the activation function decides "how active" the neuron is. Without activation, a neuron is simply a linear regression. #### activation formula and functions ```typescript z = w₁ x₁ + w₂ x₂ + … + wₙ xₙ + b output = f(z) ← activation function Activation functions: Sigmoid: f(z) = 1 / (1 + e^-z) → [0,1] historically the first, now rare ReLU: f(z) = max(0, z) → [0,∞) standard for hidden layers SiLU/Swish: f(z) = z sigmoid(z) → smooth ReLU used in modern LLM (SwiGLU) # Why nonlinearity is critical: without f(z): 100 layer network = one linear function with f(z): the network can approximate ANY function ``` ### 02.06 Why "network": layers of neurons One neuron is weak. When neurons are connected in layers, and layers are connected in series, a network emerges. Each layer transforms the input's representation: the first layer sees raw pixels, the last layer sees abstract concepts. This is called a feature hierarchy. #### Diagram ```text Two-layer network (3→4→2): Hidden layer Output layer x₁ ──┐ ┌─ h₁ ──┐ ├──▶│─ h₂ ──┤──▶ o₁ (class A) x₂ ──┤ │─ h₃ ──┤──▶ o₂ (class B) ├──▶└─ h₄ ──┘ x₃ ──┘ ↑ each arrow = one weight w Each neuron hᵢ: hᵢ = f(w_{i1} x₁ + w_{i2} x₂ + w_{i3} x₃ + bᵢ) Network parameters 3→4→2: layer 1: 3x4 + 4 = 16 (weights + biases) layer 2: 4×2 + 2 = 10 total: 26 parameters ``` #### why "neural NETWORK" - neuro - from neuron biology - network - neurons connected to each other - output of one neuron = input of the next ### 02.07 Universal approximation theorem It has been mathematically proven (Cybenko, 1989): a neural network with one hidden layer and a sufficient number of neurons can approximate any continuous function with any accuracy. This explains why neural networks are so versatile - they are not limited to a specific form of function. #### what does this mean in practice ```text Theorem: ∃ network W such that |W(x) − f(x)| < ε for any x In other words, a neural network can learn: ✓ grammar rules (no one wrote explicitly) ✓ what does “cat” mean based on thousands of photos ✓ connection between words in the text ✓ style of a specific author But: the theorem says “exists”, not “how to find” → you need enough data → training needed (backprop + optimizer) → the right architecture is needed ``` ## 03. Deep Learning: Why Depth ### 03.08 Shallow vs Deep Network: Hierarchy of Features A “deep” neural network is simply a network with many layers. But depth is not just quantity: each layer builds more abstract representations on top of the previous one. This is what allows deep networks to understand images, sound and text without manual cues. #### Diagram ```text Example: Face Recognition (CNN) Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 pixels → edges/corners → textures → parts of the face → “person” contrasts stripes nose, eyes Ivan Petrov gradients patterns cheekbones vs not a person Key: no one told the network to “look for the edges.” She herself has found that edges are a useful intermediate feature. Same for text (Transformer): bytes/characters → morphemes → words/meanings → phrases/context → intentions/facts ``` #### deep learning = automatic feature engineering - signs are learned, not invented - works on raw data: pixels, tokens, PCM - requires a lot of data and calculations - interpretability is worse than linear ones ### 03.09 How a neural network learns Learning is an iterative process of minimizing error. The model makes a prediction, compares it with the correct answer, calculates “how much it was wrong” (loss), and slightly adjusts all the weights in the right direction. #### gradient descent loop ```python ① Forward pass: prediction = model(x) ← through all layers ② Loss (error function): L = loss_fn(prediction, y_true) for example cross-entropy: how wrong is the probability? ③ Backward pass (backpropagation): gradients = ∂L/∂W ← chain rule across all layers "how to change each weight to reduce L?" ④ Update (optimizer): W ← W - lr gradients ← small step ⑤ Repeat a million times → the weights gradually “correctly” describe the data ``` ### 03.10 Why neural networks replaced the classics Until 2012, SVM and Random Forest won most benchmarks. In 2012, AlexNet (CNN) reduced the error on ImageNet from 26% to 16%—by half. Since then, neural networks have dominated wherever there is a lot of data and complex features. | | Classic ML | Neural network | | --- | --- | --- | | Signs | Hand Engineering | Automatically | | Raw data | Bad | Okay | | Data scale | Plateau at 10k–100k | Grows with data | | Interpretability | High | "Black box" | | Computations | CPU, fast | GPU, expensive | | Best problems | tabular data | text, photo, audio | ## 04. The text problem: sequences and RNNs ### 04.11 Why is text more complex than a picture? A picture is a fixed grid of pixels. The text is a sequence of variable length, where the order is critical: “the dog bit the man” ≠ “the man bit the dog.” A regular MLP does not know the concept of “before” and “after” - a special architecture is needed. #### specificity of the text ```text Problem 1 - variable length: "Yes." → 2 tokens "War and Peace" → ~580,000 tokens → MLP requires a fixed input Problem 2 - long-distance dependencies: "The neighbor's cat around the corner meowed." → “meowed” depends on “cat” after 9 words Problem 3 - order is important: "not good" ≠ "good" "I just left" ≠ "I just left" Problem 4 - context changes meaning: "key" → door? spring? way to solve? ``` ### 04.12 RNN: Recurrent Network RNN (Recurrent Neural Network) solves the problem of sequences through “memory”: it processes tokens one by one, passing the hidden state h forward. h is a “compressed memory” of everything that has been read up to this point. #### Diagram ```text Time-unfolded RNN: x₁ → [RNN] → h₁ → [RNN] → h₂ → [RNN] → h₃ → output ↑ ↑ ↑ same weights W (weight sharing) Formula: hₜ = tanh(W_h h_{t-1} + W_x xₜ + b) ↑ hidden state depends on previous h Problem: "Masha went to Paris. Does she speak ____?" t=1 t=4 t=5 t=7 t=10 to token t=10 information about “Paris” has greatly faded → vanishing gradient through a long chain ``` #### solutions on top of RNN - LSTM - gated memory cell (1997) - GRU - simplified LSTM (2014) - still bad at >500 tokens - sequential processing = cannot be parallelized ### 04.13 Bottleneck: vector bottleneck In the translation task (seq2seq), the encoder compressed an entire sentence into a single fixed-size vector. Then the decoder turned it into translation. With long sentences, the vector could not retain all the information - the quality dropped. #### Diagram ```text Seq2Seq without attention (2014): "I love cats" x₁ x₂ x₃ ↓ ↓ ↓ [enc]→[enc]→[enc] → context vector c ← ONLY ONE vector! ↓ [dec]→[dec]→[dec] "I love cats" Problem with long sentences: c ∈ ℝ^512 - cannot contain all the information in a long document Bahdanau (2015) proposed a solution: “What if the decoder itself chooses, WHICH encoder token should I look at? → this is the idea of attention ``` ## 05. Attention: the heart of modern NLP ### 05.14 Intuition Attention: “what to look for” Attention is a soft search mechanism: instead of reading all the information at once, the model learns to assign importance weights. For each position, it calculates how “relevant” each other position is right now. #### Diagram ```tex Human analogy: Question: “He went to Paris. What did she do? Reading “she” → attention to “he” (coreference) = high weight Reading “did” → attention to “left” (action) = high weight Reading “did” → attention to “Paris” = low weight Mathematically: For token q (query - “what am I looking for?”): score(q, kⱼ) = q · kⱼ ← scalar product αⱼ = softmax(score) ← normalization → weights [0,1], sum=1 out = Σⱼ αⱼ · vⱼ ← weighted sum of values An example of attention weights when translating “love” → “I love”: I love cats love [0.05 0.88 0.07] ← looks almost only at “love” I [0.91 0.05 0.04] cats [0.03 0.09 0.88] ``` #### revolutionism - direct access to any token - no attenuation - weights are calculated dynamically depending on the context - parallel to all positions - Bahdanau 2015 - first attention for NLP ### 05.15 Self-Attention: “attention to yourself” In the original attention decoder looked at the encoder. Self-attention is when a sequence looks at itself. Each token is simultaneously a query, a key, and a value. This allows you to build contextual representations without recurrence. #### mechanics Q, K, V ```tex For each token x_i: Q_i = x_i · W_Q ← “what am I looking for?” K_j = x_j · W_K ← “what am I offering?” V_j = x_j · W_V ← “what am I returning?” score(i,j) = Q_i·K_jᵀ / √d_k α_ij = softmax(score) out_i = Σⱼ α_ij · V_j Result: "key" in the context of "door" has a different vector than the “key” in the “spring” → contextual embeddings! ``` #### vs recurrence - O(1) steps between any tokens (RNN: O(n)) - parallel processing of the entire sequence - O(n²) memory - expensive on long contexts ### 05.16 Multi-Head: several types of attention One attention mechanism answers one “question”. Multi-head attention launches several parallel attentions with different weights - each head specializes in its own type of connections: syntax, coreference, semantics, position. #### what different heads teach ```text Head 1: syntactic connections “the cat meowed” - the head sees the subject↔predicate Head 2: coreference "Masha went home. She was tired." — the head notices “she” = “Masha” Head 3: Positional Patterns — monitors neighboring tokens Head 4: semantic proximity - "king" ↔ "monarch" ↔ "ruler" → concatenation of all heads → W_O → final vector → each token gets a “comprehensive” understanding of the context ``` ## 06. Transformer: «Attention Is All You Need» ### 06.17 2017: «Attention Is All You Need» Vaswani et al. (Google, 2017) proposed to remove recurrence completely and build an architecture based only on attention and MLP. The result is better quality, 3x faster training. This became the foundation for all LLMs. #### what has changed ```text Before (RNN/LSTM seq2seq): processing: sequential (token by token) long dependencies: lost training on 8 GPU: weeks context: ~200–500 tokens After (Transformer): processing: parallel (all tokens at once) long dependencies: direct O(1) access training: days context: scalable (hundreds of thousands, 1M+, for some models up to ~2M) The key idea is positional embeddings: “if there is no recurrence, how can we know the order?” → add position information to the token vector itself ``` ### 06.18 Transformer block: anatomy One transformer block is two subblocks with residual connections. Self-attention “mixes” information between positions - it recognizes the context. The MLP “thinks” about each position independently—applies knowledge. LLM = stack of such blocks. #### Diagram ```text x ──────────────────────────────────────┐ ↓ │ LayerNorm │ skip ↓ │ Multi-Head Self-Attention │ ↓ │ x' = x + Attention(x) ←─────────────────┘ x' ─────────────────────────────────────┐ ↓ │ LayerNorm │ skip ↓ │ Feed-Forward MLP (d→4d→d) │ ↓ │ x'' = x' + MLP(x') ←────────────────────┘ Attention = “what is important for the context” MLP = "what does it mean" (knowledge repository) ``` ### 06.19 Why block stack = Language Model The job of the language model is to predict the next token. A stack of transformer blocks builds an increasingly deeper representation of the context, and the final linear layer (LM Head) converts it into probabilities over a dictionary. Training in “guess the next word” gives language abilities. #### language modeling ```text Task: P(next token | all previous ones) Entrance: "Moscow - the capital" ↓ × 32 transformer blocks ↓ LM Head (d_model → vocab_size) ↓ softmax Probabilities: "Russia" → 0.71 "Russia." → 0.12 "Russia," → 0.08 "world" → 0.02 … Training: on trillions of text tokens learning to guess the next word → side effect: the model “understands” language, facts, logic ``` #### scale laws (Kaplan et al. 2020) - more parameters → better - more data → better - more compute → better - predictable - can be planned ## 07. From Transformer to LLM ### 07.20 Three ingredients of LLM LLM is not a new architecture. This is a decoder-only transformer trained in three stages on a fundamentally different scale of data and calculations. GPT-2 (2019) showed: just a large transformer can already do a lot without additional training. #### three stages of creating an LLM ```text ① Pretraining (language modeling): data: the entire Internet, books, code (~10T tokens) task: predict next token result: the model “knows the language” and the facts of the world ② Supervised Fine-Tuning (SFT): data: pairs (question → good answer), ~100k task: learning to follow instructions result: model responds as assistant ③ RLHF/DPO (alignment): data: human ranks answers task: to maximize “human preference” result: safe, useful, honest answer Pretraining: 99% calculations SFT + RLHF: 1% computation but UX critical ``` ### 07.21 Emergent Abilities: abilities from scale At a certain scale, a model suddenly exhibits abilities that smaller models did not have—not gradually, but in leaps and bounds. No one specifically taught GPT-4 to solve chemistry problems—it taught itself. #### examples of emergent abilities ```sql Appears at ~10B+ parameters: ✓ Multi-step reasoning (chain-of-thought) ✓ Arithmetic (3-5 characters) ✓ Translation into languages not from training ✓ Analogies and metaphors Appears at ~100B+ parameters: ✓ Coding in new languages ✓ Explain your reasoning ✓ Solving problems using symbolic logic ✓ Understanding sarcasm and irony Why: debatable. Possibly critical mass compressed knowledge allows you to combine them ``` ### 07.22 LLM is a neural network: the full argument Now you can collect a complete answer to the question “why LLM is a neural network.” This is not a metaphor or marketing - this is a specific architectural chain from the perceptron to the modern model. #### full argument ```sql LLM is a neural network because: "Neuro": consists of artificial neurons (weighted sum + nonlinearity) inspired by biological brain neurons "Network": neurons are connected in layers output of one → input of the next 32–80 layers of transformer blocks "Transformer": a special neural network where the “connections” between neurons are not fixed, but are calculated dynamically through the attention mechanism "Language": trained on the predict-next-token task → learned grammar, facts, logic, style Parameters: 8B–1T+ weights wᵢ, selected by gradient descent on trillions of text tokens ``` ## 08. The big picture: 70 years behind one scheme ### 08.23 Development line: from perceptron to GPT Each next step solved a specific problem of the previous one. This is not a random sequence of inventions - it is a deliberate movement towards a universal sequence processor. #### Timeline - 1958 Perceptron Rosenblatt. One neuron. Linear classifier. - 1986 Backprop Rumelhart. Deep networks can be trained. - 1997 LSTM Hochreiter. Memory in a recurrent network. - 2012 AlexNet CNN defeats ImageNet. Deep learning works. - 2015 Attention Bahdanau. Decoder looks at the required encoder tokens. - 2017 Transformer Vaswani. Only attention, no recurrence. - 2018 BERT/GPT Pretrain on a large body → finetune to the task. - 2022+ LLM Scale + RLHF → ChatGPT, Claude, Gemini. #### Diagram ```typescript Predecessor Problem → Child Solution: The linear model does not solve nonlinear problems ↓ solution: nonlinear activation function Neural network (MLP) does not work with sequences ↓ solution: recurrent connections RNN/LSTM loses long-range dependencies, cannot be parallelized ↓ solution: direct access to any position Attention (RNN) bottleneck vector, still recurrent ↓ solution: remove recurrence completely Transformer is small - doesn't know enough ↓ solution: scale + RLHF LLM ← here we are now ────────────────────────────── ────────────────────────────── Each LLM = neural NETWORK of: ~80 layers × (self-attention + MLP) + embedding + LM head with ~70,000,000,000 trained weights w each of which is a descendant of Rosenblatt's 1958 perceptron ```