{ "type": "new-runtime-knowledge-guide", "version": 1, "generated_at": "2026-07-23", "volume": 16, "slug": "classical-ml-reference", "title": "Classical Machine Learning in Plain English", "description": "A cheat sheet for talking with ML specialists in one language: what are data, features, target, loss, generalization and prediction; what problems are solved by linear models, trees, SVM, kNN, ensembles, clustering and PCA; and why the transition to neural networks did not cancel the basic mechanics of errors and optimization, but scaled it.", "track": { "slug": "ml-and-decision-systems", "title": "ML & Decision Systems", "route": "https://newruntime.com/learn/tracks/ml-and-decision-systems/", "position": 3 }, "counts": { "sections": 7, "cards": 31 }, "routes": { "html": "https://newruntime.com/learn/classical-ml-reference/", "json": "https://newruntime.com/learn/classical-ml-reference.json" }, "sections": [ { "number": "01", "title": "Foundation and vocabulary", "intro": "", "cards": [ { "number": "01", "title": "What is ML in one scheme", "tag": "Foundation", "description": "Machine Learning is a way to teach a function to make useful predictions based on examples, rather than writing all the rules manually. We have data, a target variable, and a family of models. Training selects the model parameters so that the error on known examples becomes small, and then we use the same model on new objects.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "typescript", "value": "THE MOST GENERAL ML SCHEME\n\nGiven:\n object x for example: client, letter, photo, transaction\n target y price, class, probability, risk, cluster\n\nChoose:\n model f(x; θ) linear / tree / svm / neural net / ...\n\nTraining:\n choose θ so that\n predictions f(x; θ) were close to y\n\nUsage:\n new x_new → f(x_new; θ) → y_hat\n\nML = not \"magic\", but setting up a parameterized function based on data" } ] }, { "number": "02", "title": "Data language: sample, feature, target, label", "tag": "Foundation", "description": "Most conversations in ML come down to a table. The row is the object of observation, the columns are the signs, and the target column is what we want to predict. Even if the data is not tabular by nature, they still try to bring it to such a representation.", "blocks": [ { "kind": "code", "label": "terms", "language": "text", "value": "X = feature matrix, shape: [n_samples, n_features]\ny=target/labels\n\nsample/instance/row:\n one row of data\n\nfeature:\n one measurable characteristic\n\ntarget:\n what we want to predict\n\nlabel:\n target in a classification problem" }, { "kind": "list", "label": "example", "items": [ "sample = one client", "features = age, check, city, activity", "target = will leave / will not leave" ] } ] }, { "number": "03", "title": "What it does \"prediction\" really mean?", "tag": "Foundation", "description": "Prediction in ML is not just about “guessing the future number”. The model can produce a number, class, probability, score, ranking score, embedding or cluster id. In real systems, what is often more important is not the final class, but the intermediate score, which is then cut by a threshold or goes into downstream logic.", "blocks": [ { "kind": "table", "headers": [ "Task", "Model output", "What are they doing with it?" ], "rows": [ [ "Regression", "number", "price, demand, time, risk" ], [ "Classification", "probability / class", "threshold, alert, routing" ], [ "Ranking", "score", "sorting candidates" ], [ "Clustering", "cluster id", "segmentation and analysis" ], [ "PCA / embeddings", "new vector", "visualization, retrieval, downstream model" ] ] } ] }, { "number": "04", "title": "The formula for almost everything ML: family + loss + optimizer + regularization", "tag": "Foundation", "description": "Almost any ML pipeline can be reduced to one template. You choose a family of functions, set a measure of error, sometimes add a complexity penalty, and then the optimizer looks for good parameters. Further, all engineering revolves around data quality, choice of loss, metrics, regularization and the ability of the model to generalize.", "blocks": [ { "kind": "code", "label": "generalized objective", "language": "sql", "value": "find θ that minimizes:\n\n J(θ) = (1 / N) Σ_i L(f(x_i; θ), y_i) + λΩ(θ)\n\nwhere:\n f(x; θ) = model\n L = loss using one example\n Ω(θ) = difficulty penalty\n λ = regularization strength\n\nLinear regression, logistic regression, boosting, and neural networks are built around this formula" }, { "kind": "list", "label": "typical loss", "items": [ "MSE for regression", "log loss for classification", "hinge loss for SVM style", "within-cluster sum of squares for k-means" ] } ] }, { "number": "05", "title": "Training, inference, generalization - three different modes", "tag": "Foundation", "description": "Beginners often confuse learning and using the model. During training the parameters change. During inference, the parameters are frozen and the model only calculates the output. And generalization is the main question: how well does the frozen model work not on familiar examples, but on new ones.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "THREE MODES\n\nTraining:\n X_train, y_train\n → change θ\n → reduce loss\n\nInference:\n x_new\n → θ fixed\n → get y_hat\n\nGeneralization:\n X_test, y_test\n → check if the model has learned only train\n\nA good train score without generalization is worth almost nothing" } ] } ] }, { "number": "02", "title": "How a model learns", "intro": "", "cards": [ { "number": "06", "title": "Gradient Descent: Descent by mistake", "tag": "Learning", "description": "Not all models are trained with gradient descent, but it is one of the central mechanisms of ML. The idea is simple: see how the loss changes with small changes in parameters, and shift the parameters to where the error drops. This same thinking then goes directly into neural networks.", "blocks": [ { "kind": "code", "label": "minimal mechanics", "language": "sql", "value": "init θ randomly or by heuristic\nrepeat:\n y_hat = f(X; θ)\n loss = L(y_hat, y)\n grad = ∂loss / ∂θ\n θ = θ - η * grad\n\nwhere:\n η = learning rate" }, { "kind": "list", "label": "important nuances", "items": [ "too small step = long", "too big step = fly apart", "SGD = count the gradient by batches, not by the entire dataset" ] } ] }, { "number": "07", "title": "Overfitting vs Underfitting", "tag": "Learning", "description": "Underfitting means that the model is too weak and cannot catch the structure even on train. Overfitting means the opposite: the model has adapted too well to train noise and loses quality on new data. In ML, almost everything revolves around finding a balance between these two extremes.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "INTUITION\n\nUnderfit:\n train error high\n test error high\n the model is too simple\n\nGood fit:\n train error low\n test error low\n\nOverfit:\n train error very low\n test error is noticeably worse\n the model adjusted to the noise" } ] }, { "number": "08", "title": "Regularization: penalty for excessive complexity", "tag": "Learning", "description": "Regularization is needed to prevent the model from using too “aggressive” parameters or too complex rules. In linear models these are L1/L2 weight penalties. In trees, these are restrictions on the depth and size of leaves. In boosting this is shrinkage, the number of trees and early stopping.", "blocks": [ { "kind": "code", "label": "forms of regularization", "language": "text", "value": "linear models:\n L2: + λ ||w||²\n L1: + λ ||w||₁\n\ntrees:\n max_depth\n min_samples_leaf\n pruning\n\nboosting:\n learning_rate\n max_depth\n n_estimators\n early_stopping" } ] }, { "number": "09", "title": "Train / Validation / Test and Cross-Validation", "tag": "Learning", "description": "An ML model cannot be fairly evaluated on the same data on which you selected the hyperparameters. Therefore, the data is divided into train, validation and test. When there is not enough data, they use cross-validation: they retrain the model several times on different folds and average the quality.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "HONEST RATING SCHEME\n\ntrain:\n learning model parameters\n\nvalidation:\n select hyperparameters\n comparing models\n solve threshold\n\ntest:\n one honest final shot\n\nk-fold CV:\n data is cut into k parts\n k times one part plays the role of validation\n total = average quality for all folds" }, { "kind": "list", "label": "antipatterns", "items": [ "select a model using test set", "normalize the entire dataset to split", "keep test as \"last exam\"" ] } ] } ] }, { "number": "03", "title": "Types of tasks", "intro": "", "cards": [ { "number": "10", "title": "Regression: predicting a number", "tag": "Task", "description": "Regression is a task where the target is continuous: price, delivery time, sales volume, probability of default as a number from 0 to 1, temperature, lifetime value. The main question here is not “to which class to classify”, but “how large the output will be.”", "blocks": [ { "kind": "code", "label": "examples", "language": "text", "value": "x = apartment\ny = price\n\nx = client + context\ny = expected revenue\n\nx = ticket\ny = solution time" } ] }, { "number": "11", "title": "Classification: predicting class", "tag": "Task", "description": "Classification is tasks with discrete tags: spam / not spam, fraud / not fraud, product class, text language, disease. Often, the model first produces a probability or score, and only then the system turns it into a class through a threshold.", "blocks": [ { "kind": "list", "label": "options", "items": [ "binary classification", "multiclass classification", "multilabel classification", "class imbalance is often more important than the model itself" ] } ] }, { "number": "12", "title": "Score, probability, threshold, ranking are not the same thing", "tag": "Task", "description": "In many systems, the model does not produce a final solution, but a score. This score can be calibrated to probability, cut by a threshold, or used only for ranking. Conversations about precision/recall and business trade-off almost always revolve around this layer, and not around the “model architecture”.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "DECISION LAYER\n\nmodel:\n x → score = 2.31\n\nfurther options are possible:\n sigmoid(score) → probability = 0.91\n if probability > 0.8 → alert\n sort by score desc → ranking\n\nThe same model can live in different product modes" }, { "kind": "list", "label": "consequences", "items": [ "For ranking, order is important, not calibration", "for alert systems, the threshold is adjusted to the cost of errors", "accuracy can hide a bad threshold" ] } ] }, { "number": "13", "title": "Unsupervised tasks: no target tag", "tag": "Task", "description": "In unsupervised learning there is no ready-made `y` that the model must guess. Instead, they look for structure in the data itself: groups of similar objects, low-dimensional representation, density, outliers. This is often not a “prediction” in the narrow sense, but a way to organize the feature space.", "blocks": [ { "kind": "code", "label": "typical modes", "language": "sql", "value": "clustering:\n find groups of similar objects\n\ndimensionality reduction:\n compress data into fewer coordinates\n\nanomaly detection:\n find points that are not similar to the norm\n\ndensity estimation:\n assess where data “lives” in space" } ] } ] }, { "number": "04", "title": "Basic supervised models", "intro": "", "cards": [ { "number": "14", "title": "Linear models: the most important baseline", "tag": "Model", "description": "Linear models assume that the final response can be expressed as a weighted sum of features. They are not always the most accurate, but they are almost always needed as a baseline: they are fast, understandable, cheap, easy to debug, and show well whether there is at least a simple signal in the data.", "blocks": [ { "kind": "code", "label": "linear model kernel", "language": "text", "value": "y_hat = w₀ + w₁x₁ + w₂x₂ + ... + w_px_p\n\nLinear Regression:\n minimizes MSE / residual sum of squares\n\nRidge:\n the same linear model + L2 penalty\n\nLasso:\n the same linear model + L1 penalty\n may nullify some signs" }, { "kind": "list", "label": "when you are strong", "items": [ "clear numeric/tabular data", "interpretable baseline", "difficult to catch complex nonlinearities without feature engineering" ] } ] }, { "number": "15", "title": "Logistic Regression: linear classification via probability", "tag": "Model", "description": "Despite the name, logistic regression solves the classification problem. It builds a linear score, then passes it through the logistic function and gets the class probability. Therefore, this is one of the most standard and strong baselines for binary classification.", "blocks": [ { "kind": "code", "label": "essence", "language": "python", "value": "score = w x + b\np(y=1|x) = sigmoid(score)\n\nif p > threshold:\n class = 1\notherwise:\n class = 0\n\nloss:\n log loss / cross-entropy" } ] }, { "number": "16", "title": "k-Nearest Neighbors: predict by neighbors", "tag": "Model", "description": "kNN teaches almost nothing in the classical sense: it stores train examples and, with a new request, looks for the nearest objects in the feature space. For classification it does voting, for regression it averages the target neighbors. This is a very intuitive, but sensitive to the scale of features and the size of the dataset.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "IDEA kNN\n\nnew point x_new\n ↓\nfind k nearest train points\n ↓\nclassification:\n majority vote\n\nregression:\n average target\n\nmodel = feature space geometry" } ] }, { "number": "17", "title": "Naive Bayes: fast probabilistic approximation", "tag": "Model", "description": "Naive Bayes relies on the Bayes theorem and a rough but convenient assumption: features are conditionally independent given a fixed class. This assumption is often literally false, but the model can still perform surprisingly well, especially in text classification and as a very fast baseline.", "blocks": [ { "kind": "code", "label": "idea", "language": "text", "value": "P(y | x₁...x_n) ∝ P(y) * Π_i P(x_i | y)\n\n\"naive\" = we consider the signs to be independent\n\npros:\n fast fit\n fast predict\n good on sparse text\n\ncons:\n strong assumption of independence" } ] }, { "number": "18", "title": "SVM: find the dividing border with maximum margin", "tag": "Model", "description": "Support Vector Machine builds a dividing surface between classes so that the distance to the nearest training examples is maximum. In the linear case, this is a neat geometric model of the boundary. With the kernel trick, SVM can also work with nonlinear partitions, although it scales worse than modern tabular ensembles.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "INTUITION MARGIN\n\nclass A | class B\n ● ● ● | ○ ○ ○\n ● ● ● | ○ ○ ○\n\ntask:\n draw the line like this\n so that the closest points on both sides\n were as far away as possible\n\nthese closest points = support vectors" } ] }, { "number": "19", "title": "Decision Tree: if-else rules learned from data", "tag": "Model", "description": "Decision tree builds a tree of conditions like `if age < 30 and income > 100k`. At each node, it selects the partition that best reduces impurity or error. Trees are easy to read, they are able to catch nonlinearities and interactions of features, but alone they are unstable and prone to overfit.", "blocks": [ { "kind": "code", "label": "how does a tree think", "language": "python", "value": "if feature_7 < 13.5:\n go left\nelse:\n go right\n\nrepeat until sheet\n\nleaf:\n class = majority label\n or\n value = average target" } ] } ] }, { "number": "05", "title": "Ensembles and unsupervised models", "intro": "", "cards": [ { "number": "20", "title": "Random Forest / Bagging: many trees instead of one", "tag": "Ensemble", "description": "Bagging reduces variance by averaging across many highly variable models. Random Forest is bagging over trees: each tree learns from a bootstrap subsample and a random subset of features. As a result, a forest is usually more stable and stronger than a single tree.", "blocks": [ { "kind": "code", "label": "random forest scheme", "language": "python", "value": "for t in 1..T:\n sample data with replacement\n sample subset of features\n fit deep decision tree\n\nclassification:\n final class = majority vote\n\nregression:\n final value = average of trees" } ] }, { "number": "21", "title": "Gradient Boosting: Correct the mistakes of previous trees", "tag": "Ensemble", "description": "Boosting is not built as parallel voting, but as sequential error correction. Each new tree tries to improve where the current ensemble goes wrong. This is why gradient boosting and its successors often become the best models on tabular data with good feature preparation.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "INTUITION BOOSTING\n\nmodel_1:\n rude first answer\n\nmodel_2:\n looks where model_1 made a mistake\n and adds an amendment\n\nmodel_3:\n corrects the remainder after the first two\n\nresult:\n final_prediction =\n base + correction_1 + correction_2 + ..." }, { "kind": "list", "label": "practically", "items": [ "often the best choice for tabular data", "XGBoost / LightGBM / CatBoost - industry standard", "requires careful tuning and leakage control" ] } ] }, { "number": "22", "title": "K-Means: partition points into K compact groups", "tag": "Unsupervised", "description": "K-means looks for `K` centers and assigns each point to the nearest one. Then the centers are recalculated as the average of their points, and the cycle is repeated. This is a simple and very popular method of segmentation if the clusters are approximately “spherical” and the number of groups is known in advance.", "blocks": [ { "kind": "code", "label": "k-means iteration", "language": "sql", "value": "1. select K centers\n2. assign each point the nearest center\n3. recalculate the center as the mean of the cluster\n4. repeat until stabilized\n\ngoal:\n minimize within-cluster sum of squares" } ] }, { "number": "23", "title": "DBSCAN: clusters as dense regions, noise separately", "tag": "Unsupervised", "description": "DBSCAN does not require the number of clusters to be specified in advance. It looks at where the points are dense enough and considers such areas to be clusters. Objects that are not adjacent to any dense area are marked as noise. This is useful when clusters have complex shapes.", "blocks": [ { "kind": "list", "label": "when it's good", "items": [ "free-form clusters", "you need to catch the noise separately", "worse at very different densities" ] } ] }, { "number": "24", "title": "PCA: compress data while preserving maximum variation", "tag": "Unsupervised", "description": "Principal Component Analysis looks for new axes in the data along which variation is greatest. Typically the first few components contain most of the \"structure\" of the data set. Therefore, PCA is used for visualization, dimensionality reduction, denoising, and feature preparation.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "INTUITION PCA\n\nwas:\n 100 signs\n\nfound:\n new axes c1, c2, c3...\n which explain the maximum variance\n\nleft:\n only the first 10 components\n\nreceived:\n compact presentation without some of the noise" } ] } ] }, { "number": "06", "title": "Practice and workflow", "intro": "", "cards": [ { "number": "25", "title": "Feature Engineering and preprocessing are half the model", "tag": "Workflow", "description": "In classical ML, the quality of features is often more important than the choice of algorithm. Normalization, one-hot encoding, gap handling, time aggregations, logarithms, feature interactions, target encoding and domain features can give a greater gain than changing the model from logistic regression to something “smarter”.", "blocks": [ { "kind": "code", "label": "typical preprocessing", "language": "text", "value": "numeric:\n impute missing\n scale/standardize\n\ncategorical:\n one-hot/target encoding\n\ntime:\n lags, rolling stats, calendar features\n\ntext:\n bag-of-words/TF-IDF/embeddings" } ] }, { "number": "26", "title": "Metrics: the model should be good at what you really care about", "tag": "Workflow", "description": "Loss, which the model optimizes, and the metric by which the business makes a decision do not have to coincide. For regression, large errors may be important to you, and for classification, the prices of false positive and false negative are asymmetrical. Therefore, a conversation about quality always begins with the question: which error is more expensive and what exactly do we want to optimize.", "blocks": [ { "kind": "table", "headers": [ "Task type", "Metrics", "What is measured" ], "rows": [ [ "Regression", "MSE, RMSE, MAE, R²", "average error size, variance explained" ], [ "Classification", "accuracy", "percentage of correct answers" ], [ "Imbalanced classification", "precision, recall, F1, PR-AUC", "balance of false positives and omissions" ], [ "Probabilities", "log loss, Brier score", "how good are the probabilities?" ], [ "Ranking", "NDCG, MAP, AUC", "how good is the order of objects" ] ] } ] }, { "number": "27", "title": "Pipeline and data leakage", "tag": "Workflow", "description": "One of the most common mistakes is to first convert the entire dataset and then divide it into train/test. This is how the model accidentally learns information from the future test. The correct pipeline ensures that any fit occurs only on the train fold, and then the same transform parameters are applied to validation/test.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "BAD vs GOOD\n\nBad:\n scale(all data)\n split(train, test)\n ← leakage\n\nOkay:\n split(train, test)\n fit scaler on train\n transform train/test with train-scaler\n fit model on train" } ] }, { "number": "28", "title": "Normal workflow: baseline → error analysis → tuning", "tag": "Workflow", "description": "An experienced ML process almost never starts with the most complex model. First, a simple baseline is built to understand the signal level in the data. Then they look at errors, fix symptoms and leakage, and only then tune the hyperparameters or move on to more powerful models.", "blocks": [ { "kind": "code", "label": "operating procedure", "language": "text", "value": "1. understand the task and metric\n2. make a simple baseline\n3. check split / leakage / preprocessing\n4. do error analysis\n5. improve features\n6. compare strong models\n7. tune hyperparameters\n8. think about sales and monitoring" } ] } ] }, { "number": "07", "title": "Bridge to neural networks", "intro": "", "cards": [ { "number": "29", "title": "Perceptron: where classic ML begins to become a neural network", "tag": "Bridge", "description": "Perceptron is a very simple linear classification unit: it sums features with weights, adds bias and applies threshold activation. In fact, this is already a model of the form `score = w x + b`, only with a nonlinear decision rule. Historically, this is one of the bridges between linear models and neural networks.", "blocks": [ { "kind": "code", "label": "perceptron formula", "language": "python", "value": "score = w x + b\n\nif score > 0:\n y_hat = 1\nelse:\n y_hat = 0\n\nupdate:\n if error:\n w = w + η (y - y_hat) x" } ] }, { "number": "30", "title": "Backpropagation: the same conversation about the error, but through many layers", "tag": "Bridge", "description": "The key transition to neural networks did not occur because “different mathematics” suddenly appeared in them. The main idea remains the same: there is loss, there are parameters, we need to reduce the error. The novelty of backprop is that it can effectively distribute the error contribution back through many intermediate layers according to the chain rule.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "FROM CLASSICAL ML TO BACKPROP\n\nclassic scheme:\n x → model → y_hat\n compare with y\n update parameters\n\nneural network:\n x → layer1 → layer2 → ... → layerN → y_hat\n compare with y\n error goes back through all layers\n each weight gets its own piece of the gradient\n\ngeneral:\n the same error minimization logic\n\nnew:\n the model itself learns intermediate representations" }, { "kind": "code", "label": "very short", "language": "text", "value": "forward pass:\n count y_hat\n\nloss:\n measure error\n\nbackward pass:\n find ∂loss / ∂w for all weights\n\nupdate:\n w = w - η * gradient" } ] }, { "number": "31", "title": "What neural networks have changed and what remains the same", "tag": "Bridge", "description": "Neural networks have changed, first of all, the power of the function and the way of working with features: now the model itself learns the representations, and does not depend so much on manual feature engineering. But the basic ML language has not gone away. There are still data, loss, train/val/test, overfitting, regularization, gradient descent and metrics.", "blocks": [ { "kind": "table", "headers": [ "Remains the same", "Has changed a lot" ], "rows": [ [ "X, y, loss, metrics, generalization", "scale of parameters and calculations" ], [ "train / val / test", "the model itself teaches the signs" ], [ "error optimization", "deep multi-layer views" ], [ "regularization and fight against overfit", "new architectures: CNN, RNN, Transformer" ] ] } ] } ] } ] }