Classical Machine Learning in Plain English

Machine Learning is a way to teach a function to make useful predictions based on examples, rather than writing all the rules manually. We have data, a target variable, and a family of models. Training selects the model parameters so that the error on known examples becomes small, and then we use the same model on new objects.

THE MOST GENERAL ML SCHEME

Given:
  object x for example: client, letter, photo, transaction
  target y price, class, probability, risk, cluster

Choose:
  model f(x; θ) linear / tree / svm / neural net / ...

Training:
  choose θ so that
  predictions f(x; θ) were close to y

Usage:
  new x_new → f(x_new; θ) → y_hat

ML = not "magic", but setting up a parameterized function based on data

Most conversations in ML come down to a table. The row is the object of observation, the columns are the signs, and the target column is what we want to predict. Even if the data is not tabular by nature, they still try to bring it to such a representation.

X = feature matrix, shape: [n_samples, n_features]
y=target/labels

sample/instance/row:
  one row of data

feature:
  one measurable characteristic

target:
  what we want to predict

label:
  target in a classification problem

Prediction in ML is not just about “guessing the future number”. The model can produce a number, class, probability, score, ranking score, embedding or cluster id. In real systems, what is often more important is not the final class, but the intermediate score, which is then cut by a threshold or goes into downstream logic.

Almost any ML pipeline can be reduced to one template. You choose a family of functions, set a measure of error, sometimes add a complexity penalty, and then the optimizer looks for good parameters. Further, all engineering revolves around data quality, choice of loss, metrics, regularization and the ability of the model to generalize.

find θ that minimizes:

  J(θ) = (1 / N) Σ_i L(f(x_i; θ), y_i) + λΩ(θ)

where:
  f(x; θ) = model
  L = loss using one example
  Ω(θ) = difficulty penalty
  λ = regularization strength

Linear regression, logistic regression, boosting, and neural networks are built around this formula

Beginners often confuse learning and using the model. During training the parameters change. During inference, the parameters are frozen and the model only calculates the output. And generalization is the main question: how well does the frozen model work not on familiar examples, but on new ones.

THREE MODES

Training:
  X_train, y_train
  → change θ
  → reduce loss

Inference:
  x_new
  → θ fixed
  → get y_hat

Generalization:
  X_test, y_test
  → check if the model has learned only train

A good train score without generalization is worth almost nothing

Not all models are trained with gradient descent, but it is one of the central mechanisms of ML. The idea is simple: see how the loss changes with small changes in parameters, and shift the parameters to where the error drops. This same thinking then goes directly into neural networks.

init θ randomly or by heuristic
repeat:
  y_hat = f(X; θ)
  loss = L(y_hat, y)
  grad = ∂loss / ∂θ
  θ = θ - η * grad

where:
  η = learning rate

Underfitting means that the model is too weak and cannot catch the structure even on train. Overfitting means the opposite: the model has adapted too well to train noise and loses quality on new data. In ML, almost everything revolves around finding a balance between these two extremes.

INTUITION

Underfit:
  train error high
  test error high
  the model is too simple

Good fit:
  train error low
  test error low

Overfit:
  train error very low
  test error is noticeably worse
  the model adjusted to the noise

Regularization is needed to prevent the model from using too “aggressive” parameters or too complex rules. In linear models these are L1/L2 weight penalties. In trees, these are restrictions on the depth and size of leaves. In boosting this is shrinkage, the number of trees and early stopping.

linear models:
  L2: + λ ||w||²
  L1: + λ ||w||₁

trees:
  max_depth
  min_samples_leaf
  pruning

boosting:
  learning_rate
  max_depth
  n_estimators
  early_stopping

An ML model cannot be fairly evaluated on the same data on which you selected the hyperparameters. Therefore, the data is divided into train, validation and test. When there is not enough data, they use cross-validation: they retrain the model several times on different folds and average the quality.

HONEST RATING SCHEME

train:
  learning model parameters

validation:
  select hyperparameters
  comparing models
  solve threshold

test:
  one honest final shot

k-fold CV:
  data is cut into k parts
  k times one part plays the role of validation
  total = average quality for all folds

Regression is a task where the target is continuous: price, delivery time, sales volume, probability of default as a number from 0 to 1, temperature, lifetime value. The main question here is not “to which class to classify”, but “how large the output will be.”

x = apartment
y = price

x = client + context
y = expected revenue

x = ticket
y = solution time

Classification is tasks with discrete tags: spam / not spam, fraud / not fraud, product class, text language, disease. Often, the model first produces a probability or score, and only then the system turns it into a class through a threshold.

In many systems, the model does not produce a final solution, but a score. This score can be calibrated to probability, cut by a threshold, or used only for ranking. Conversations about precision/recall and business trade-off almost always revolve around this layer, and not around the “model architecture”.

DECISION LAYER

model:
  x → score = 2.31

further options are possible:
  sigmoid(score) → probability = 0.91
  if probability > 0.8 → alert
  sort by score desc → ranking

The same model can live in different product modes

In unsupervised learning there is no ready-made `y` that the model must guess. Instead, they look for structure in the data itself: groups of similar objects, low-dimensional representation, density, outliers. This is often not a “prediction” in the narrow sense, but a way to organize the feature space.

clustering:
  find groups of similar objects

dimensionality reduction:
  compress data into fewer coordinates

anomaly detection:
  find points that are not similar to the norm

density estimation:
  assess where data “lives” in space

Linear models assume that the final response can be expressed as a weighted sum of features. They are not always the most accurate, but they are almost always needed as a baseline: they are fast, understandable, cheap, easy to debug, and show well whether there is at least a simple signal in the data.

y_hat = w₀ + w₁x₁ + w₂x₂ + ... + w_px_p

Linear Regression:
  minimizes MSE / residual sum of squares

Ridge:
  the same linear model + L2 penalty

Lasso:
  the same linear model + L1 penalty
  may nullify some signs

Despite the name, logistic regression solves the classification problem. It builds a linear score, then passes it through the logistic function and gets the class probability. Therefore, this is one of the most standard and strong baselines for binary classification.

score = w x + b
p(y=1|x) = sigmoid(score)

if p > threshold:
  class = 1
otherwise:
  class = 0

loss:
  log loss / cross-entropy

kNN teaches almost nothing in the classical sense: it stores train examples and, with a new request, looks for the nearest objects in the feature space. For classification it does voting, for regression it averages the target neighbors. This is a very intuitive, but sensitive to the scale of features and the size of the dataset.

IDEA kNN

new point x_new
  ↓
find k nearest train points
  ↓
classification:
  majority vote

regression:
  average target

model = feature space geometry

Naive Bayes relies on the Bayes theorem and a rough but convenient assumption: features are conditionally independent given a fixed class. This assumption is often literally false, but the model can still perform surprisingly well, especially in text classification and as a very fast baseline.

P(y | x₁...x_n) ∝ P(y) * Π_i P(x_i | y)

"naive" = we consider the signs to be independent

pros:
  fast fit
  fast predict
  good on sparse text

cons:
  strong assumption of independence

Support Vector Machine builds a dividing surface between classes so that the distance to the nearest training examples is maximum. In the linear case, this is a neat geometric model of the boundary. With the kernel trick, SVM can also work with nonlinear partitions, although it scales worse than modern tabular ensembles.

INTUITION MARGIN

class A |    class B
  ● ● ● |    ○ ○ ○
  ● ● ● |    ○ ○ ○

task:
  draw the line like this
  so that the closest points on both sides
  were as far away as possible

these closest points = support vectors

Decision tree builds a tree of conditions like `if age < 30 and income > 100k`. At each node, it selects the partition that best reduces impurity or error. Trees are easy to read, they are able to catch nonlinearities and interactions of features, but alone they are unstable and prone to overfit.

if feature_7 < 13.5:
  go left
else:
  go right

repeat until sheet

leaf:
  class = majority label
  or
  value = average target

Bagging reduces variance by averaging across many highly variable models. Random Forest is bagging over trees: each tree learns from a bootstrap subsample and a random subset of features. As a result, a forest is usually more stable and stronger than a single tree.

for t in 1..T:
  sample data with replacement
  sample subset of features
  fit deep decision tree

classification:
  final class = majority vote

regression:
  final value = average of trees

Boosting is not built as parallel voting, but as sequential error correction. Each new tree tries to improve where the current ensemble goes wrong. This is why gradient boosting and its successors often become the best models on tabular data with good feature preparation.

INTUITION BOOSTING

model_1:
  rude first answer

model_2:
  looks where model_1 made a mistake
  and adds an amendment

model_3:
  corrects the remainder after the first two

result:
  final_prediction =
    base + correction_1 + correction_2 + ...

K-means looks for `K` centers and assigns each point to the nearest one. Then the centers are recalculated as the average of their points, and the cycle is repeated. This is a simple and very popular method of segmentation if the clusters are approximately “spherical” and the number of groups is known in advance.

1. select K centers
2. assign each point the nearest center
3. recalculate the center as the mean of the cluster
4. repeat until stabilized

goal:
  minimize within-cluster sum of squares

DBSCAN does not require the number of clusters to be specified in advance. It looks at where the points are dense enough and considers such areas to be clusters. Objects that are not adjacent to any dense area are marked as noise. This is useful when clusters have complex shapes.

Principal Component Analysis looks for new axes in the data along which variation is greatest. Typically the first few components contain most of the "structure" of the data set. Therefore, PCA is used for visualization, dimensionality reduction, denoising, and feature preparation.

INTUITION PCA

was:
  100 signs

found:
  new axes c1, c2, c3...
  which explain the maximum variance

left:
  only the first 10 components

received:
  compact presentation without some of the noise

In classical ML, the quality of features is often more important than the choice of algorithm. Normalization, one-hot encoding, gap handling, time aggregations, logarithms, feature interactions, target encoding and domain features can give a greater gain than changing the model from logistic regression to something “smarter”.

numeric:
  impute missing
  scale/standardize

categorical:
  one-hot/target encoding

time:
  lags, rolling stats, calendar features

text:
  bag-of-words/TF-IDF/embeddings

Loss, which the model optimizes, and the metric by which the business makes a decision do not have to coincide. For regression, large errors may be important to you, and for classification, the prices of false positive and false negative are asymmetrical. Therefore, a conversation about quality always begins with the question: which error is more expensive and what exactly do we want to optimize.

One of the most common mistakes is to first convert the entire dataset and then divide it into train/test. This is how the model accidentally learns information from the future test. The correct pipeline ensures that any fit occurs only on the train fold, and then the same transform parameters are applied to validation/test.

BAD vs GOOD

Bad:
  scale(all data)
  split(train, test)
  ← leakage

Okay:
  split(train, test)
  fit scaler on train
  transform train/test with train-scaler
  fit model on train

An experienced ML process almost never starts with the most complex model. First, a simple baseline is built to understand the signal level in the data. Then they look at errors, fix symptoms and leakage, and only then tune the hyperparameters or move on to more powerful models.

1. understand the task and metric
2. make a simple baseline
3. check split / leakage / preprocessing
4. do error analysis
5. improve features
6. compare strong models
7. tune hyperparameters
8. think about sales and monitoring

Perceptron is a very simple linear classification unit: it sums features with weights, adds bias and applies threshold activation. In fact, this is already a model of the form `score = w x + b`, only with a nonlinear decision rule. Historically, this is one of the bridges between linear models and neural networks.

score = w x + b

if score > 0:
  y_hat = 1
else:
  y_hat = 0

update:
  if error:
    w = w + η (y - y_hat) x

The key transition to neural networks did not occur because “different mathematics” suddenly appeared in them. The main idea remains the same: there is loss, there are parameters, we need to reduce the error. The novelty of backprop is that it can effectively distribute the error contribution back through many intermediate layers according to the chain rule.

FROM CLASSICAL ML TO BACKPROP

classic scheme:
  x → model → y_hat
  compare with y
  update parameters

neural network:
  x → layer1 → layer2 → ... → layerN → y_hat
  compare with y
  error goes back through all layers
  each weight gets its own piece of the gradient

general:
  the same error minimization logic

new:
  the model itself learns intermediate representations

forward pass:
  count y_hat

loss:
  measure error

backward pass:
  find ∂loss / ∂w for all weights

update:
  w = w - η * gradient

Neural networks have changed, first of all, the power of the function and the way of working with features: now the model itself learns the representations, and does not depend so much on manual feature engineering. But the basic ML language has not gone away. There are still data, loss, train/val/test, overfitting, regularization, gradient descent and metrics.