From a Neuron to an LLM

Normal programming:
  rules = person writes by hand
  output = rules(input)
  → works as long as there are few rules and they are clear

Machine learning:
  model = fit(inputs, outputs) ← from data!
  output = model(new_input)
  → rules are found by the algorithm itself

Deep learning (ML subtype):
  model = very flexible function (neural network)
  → finds rules even where
     a person cannot formulate them

Linear model:
  ŷ = w₁ x₁ + w₂ x₂ + … + wₙ xₙ + b
  ↑ each sign is multiplied by weight

Feature space (2D):

  x₂ ▲ ○ ○ ○ ← class A
     │ ○ ○ ○
     │ ╱ ○
     │ ╱ ← straight boundary
     │ ╱ ● ●
     │╱ ● ● ← class B
     └──────────────▶ x₁

Problem:
  x₂ ▲ ○ ● ○ ← XOR: no straight line!
     │ ● ○ ●
     └──────────────▶ x₁
  Linear model → 50% accuracy (random)

# Task: determine the sentiment of the review

Hand signs (2000s):
  "good" → +1, "bad" → -1
  number of exclamation marks → feature
  TF-IDF by dictionary → sparse vector 50k
  sentence length, capital letters...

Problem 1: "not bad" = "good"?
Problem 2: "Test!" — positive or sarcasm?
Problem 3: new words, slang → not in the dictionary

→ a person cannot come up with ALL the necessary signs
→ the neural network finds them itself from the data

Biological neuron:

  dendrite₁ ──signal──┐
  dendrite₂ ──signal──┤
  dendrite₃ ──signal──┤──▶ [cell body] ──▶ axon ──▶ next neuron
  dendrite₄ ──signal──┘ sums if sum > threshold

Artificial Neuron (Perceptron, 1958):

  x₁ ──w₁──┐
  x₂ ──w₂──┤
  x₃ ──w₃──┤──▶ Σ(wᵢ·xᵢ) + b ──▶ f(z) ──▶ output
  x₄ ──w₄──┘ weighted sum function
                                       activation

z = w₁ x₁ + w₂ x₂ + … + wₙ xₙ + b
output = f(z) ← activation function

Activation functions:

Sigmoid: f(z) = 1 / (1 + e^-z) → [0,1]
          historically the first, now rare

ReLU: f(z) = max(0, z) → [0,∞)
          standard for hidden layers

SiLU/Swish: f(z) = z sigmoid(z) → smooth ReLU
          used in modern LLM (SwiGLU)

# Why nonlinearity is critical:
without f(z): 100 layer network = one linear function
with f(z): the network can approximate ANY function

Two-layer network (3→4→2):

           Hidden layer Output layer
  x₁ ──┐ ┌─ h₁ ──┐
       ├──▶│─ h₂ ──┤──▶ o₁ (class A)
  x₂ ──┤ │─ h₃ ──┤──▶ o₂ (class B)
       ├──▶└─ h₄ ──┘
  x₃ ──┘
  ↑ each arrow = one weight w

Each neuron hᵢ:
  hᵢ = f(w_{i1} x₁ + w_{i2} x₂ + w_{i3} x₃ + bᵢ)

Network parameters 3→4→2:
  layer 1: 3x4 + 4 = 16 (weights + biases)
  layer 2: 4×2 + 2 = 10
  total: 26 parameters

Theorem: ∃ network W such that
  |W(x) − f(x)| < ε for any x

In other words, a neural network can learn:
  ✓ grammar rules (no one wrote explicitly)
  ✓ what does “cat” mean based on thousands of photos
  ✓ connection between words in the text
  ✓ style of a specific author

But: the theorem says “exists”, not “how to find”
  → you need enough data
  → training needed (backprop + optimizer)
  → the right architecture is needed

Example: Face Recognition (CNN)

Layer 1 Layer 2 Layer 3 Layer 4 Layer 5
pixels → edges/corners → textures → parts of the face → “person”
          contrasts stripes nose, eyes Ivan Petrov
          gradients patterns cheekbones vs
                                                 not a person

Key: no one told the network to “look for the edges.”
She herself has found that edges are a useful intermediate feature.

Same for text (Transformer):
bytes/characters → morphemes → words/meanings → phrases/context → intentions/facts

① Forward pass:
  prediction = model(x) ← through all layers

② Loss (error function):
  L = loss_fn(prediction, y_true)
  for example cross-entropy: how wrong is the probability?

③ Backward pass (backpropagation):
  gradients = ∂L/∂W ← chain rule across all layers
  "how to change each weight to reduce L?"

④ Update (optimizer):
  W ← W - lr gradients ← small step

⑤ Repeat a million times
  → the weights gradually “correctly” describe the data

Problem 1 - variable length:
  "Yes." → 2 tokens
  "War and Peace" → ~580,000 tokens
  → MLP requires a fixed input

Problem 2 - long-distance dependencies:
  "The neighbor's cat around the corner meowed."
  → “meowed” depends on “cat” after 9 words

Problem 3 - order is important:
  "not good" ≠ "good"
  "I just left" ≠ "I just left"

Problem 4 - context changes meaning:
  "key" → door?  spring?  way to solve?

Time-unfolded RNN:

  x₁ → [RNN] → h₁ → [RNN] → h₂ → [RNN] → h₃ → output
         ↑ ↑ ↑
       same weights W (weight sharing)

Formula:
  hₜ = tanh(W_h h_{t-1} + W_x xₜ + b)
  ↑ hidden state depends on previous h

Problem:
  "Masha went to Paris. Does she speak ____?"
   t=1 t=4 t=5 t=7 t=10
  to token t=10 information about “Paris” has greatly faded
  → vanishing gradient through a long chain

Seq2Seq without attention (2014):

  "I love cats"
     x₁ x₂ x₃
      ↓ ↓ ↓
  [enc]→[enc]→[enc] → context vector c ← ONLY ONE vector!
                           ↓
                      [dec]→[dec]→[dec]
                       "I love cats"

Problem with long sentences:
  c ∈ ℝ^512 - cannot contain
  all the information in a long document

Bahdanau (2015) proposed a solution:
  “What if the decoder itself chooses,
   WHICH encoder token should I look at?
  → this is the idea of attention

Human analogy:
  Question: “He went to Paris. What did she do?
  Reading “she” → attention to “he” (coreference) = high weight
  Reading “did” → attention to “left” (action) = high weight
  Reading “did” → attention to “Paris” = low weight

Mathematically:

  For token q (query - “what am I looking for?”):
    score(q, kⱼ) = q · kⱼ ← scalar product
    αⱼ = softmax(score) ← normalization → weights [0,1], sum=1
    out = Σⱼ αⱼ · vⱼ ← weighted sum of values

An example of attention scales when translating “love” → “I love”:

        I love cats
  love [0.05 0.88 0.07] ← looks almost only at “love”
  I [0.91 0.05 0.04]
  cats [0.03 0.09 0.88]

For each token x_i:
Q_i = x_i · W_Q ← “what am I looking for?”
K_j = x_j · W_K ← “what am I offering?”
V_j = x_j · W_V ← “what am I returning?”

score(i,j) = Q_i·K_jᵀ / √d_k
α_ij = softmax(score)
out_i = Σⱼ α_ij · V_j

Result: "key" in the context of "door"
has a different vector than the “key” in the “spring”
→ contextual embeddings!

Head 1: syntactic connections
  “the cat meowed” - the head sees the subject↔predicate

Head 2: coreference
  "Masha went home. She was tired."
  — the head notices “she” = “Masha”

Head 3: Positional Patterns
  — monitors neighboring tokens

Head 4: semantic proximity
  - "king" ↔ "monarch" ↔ "ruler"

→ concatenation of all heads → W_O → final vector
→ each token gets a “comprehensive” understanding of the context

Before (RNN/LSTM seq2seq):
  processing: sequential (token by token)
  long dependencies: lost
  training on 8 GPU: weeks
  context: ~200–500 tokens

After (Transformer):
  processing: parallel (all tokens at once)
  long dependencies: direct O(1) access
  training: days
  context: scalable (hundreds of thousands, 1M+, for some models up to ~2M)

The key idea is positional embeddings:
  “if there is no recurrence, how can we know the order?”
  → add position information to the token vector itself

x ──────────────────────────────────────┐
  ↓ │
LayerNorm │ skip
  ↓ │
Multi-Head Self-Attention │
  ↓ │
x' = x + Attention(x) ←─────────────────┘

x' ─────────────────────────────────────┐
  ↓ │
LayerNorm │ skip
  ↓ │
Feed-Forward MLP (d→4d→d) │
  ↓ │
x'' = x' + MLP(x') ←────────────────────┘

Attention = “what is important for the context”
MLP = "what does it mean" (knowledge repository)

Task: P(next token | all previous ones)

Entrance: "Moscow - the capital"
  ↓ × 32 transformer blocks
  ↓ LM Head (d_model → vocab_size)
  ↓ softmax
Probabilities:
  "Russia" → 0.71
  "Russia." → 0.12
  "Russia," → 0.08
  "world" → 0.02 …

Training: on trillions of text tokens
learning to guess the next word → side
effect: the model “understands” language, facts, logic

① Pretraining (language modeling):
  data: the entire Internet, books, code (~10T tokens)
  task: predict next token
  result: the model “knows the language” and the facts of the world

② Supervised Fine-Tuning (SFT):
  data: pairs (question → good answer), ~100k
  task: learning to follow instructions
  result: model responds as assistant

③ RLHF/DPO (alignment):
  data: human ranks answers
  task: to maximize “human preference”
  result: safe, useful, honest answer

Pretraining: 99% calculations
SFT + RLHF: 1% computation but UX critical

Appears at ~10B+ parameters:
✓ Multi-step reasoning (chain-of-thought)
✓ Arithmetic (3-5 characters)
✓ Translation into languages not from training
✓ Analogies and metaphors

Appears at ~100B+ parameters:
✓ Coding in new languages
✓ Explain your reasoning
✓ Solving problems using symbolic logic
✓ Understanding sarcasm and irony

Why: debatable. Possibly critical mass
compressed knowledge allows you to combine them

LLM is a neural network because:

"Neuro":
  consists of artificial neurons
  (weighted sum + nonlinearity)
  inspired by biological brain neurons

"Network":
  neurons are connected in layers
  output of one → input of the next
  32–80 layers of transformer blocks

"Transformer":
  a special neural network where the “connections” between
  neurons are not fixed, but are calculated
  dynamically through the attention mechanism

"Language":
  trained on the predict-next-token task
  → learned grammar, facts, logic, style

Parameters: 8B–1T+ weights wᵢ,
  selected by gradient descent
  on trillions of text tokens

Predecessor Problem → Child Solution:

The linear model does not solve nonlinear problems
      ↓ solution: nonlinear activation function
Neural network (MLP) does not work with sequences
      ↓ solution: recurrent connections
RNN/LSTM loses long-range dependencies, cannot be parallelized
      ↓ solution: direct access to any position
Attention (RNN) bottleneck vector, still recurrent
      ↓ solution: remove recurrence completely
Transformer is small - doesn't know enough
      ↓ solution: scale + RLHF
LLM ← here we are now

────────────────────────────── ──────────────────────────────
Each LLM = neural NETWORK of:
  ~80 layers × (self-attention + MLP) + embedding + LM head
  with ~70,000,000,000 trained weights w
  each of which is a descendant of Rosenblatt's 1958 perceptron