Classical Machine Learning in Plain English

Machine Learning is a way to teach a function to make useful predictions based on examples, rather than writing all the rules manually. We have data, a target variable, and a family of models. Training selects the model parameters so that the error on known examples becomes small, and then we use the same model on new objects.

Most conversations in ML come down to a table. The row is the object of observation, the columns are the signs, and the target column is what we want to predict. Even if the data is not tabular by nature, they still try to bring it to such a representation.

Prediction in ML is not just about “guessing the future number”. The model can produce a number, class, probability, score, ranking score, embedding or cluster id. In real systems, what is often more important is not the final class, but the intermediate score, which is then cut by a threshold or goes into downstream logic.

Almost any ML pipeline can be reduced to one template. You choose a family of functions, set a measure of error, sometimes add a complexity penalty, and then the optimizer looks for good parameters. Further, all engineering revolves around data quality, choice of loss, metrics, regularization and the ability of the model to generalize.

Beginners often confuse learning and using the model. During training the parameters change. During inference, the parameters are frozen and the model only calculates the output. And generalization is the main question: how well does the frozen model work not on familiar examples, but on new ones.

Not all models are trained with gradient descent, but it is one of the central mechanisms of ML. The idea is simple: see how the loss changes with small changes in parameters, and shift the parameters to where the error drops. This same thinking then goes directly into neural networks.

Underfitting means that the model is too weak and cannot catch the structure even on train. Overfitting means the opposite: the model has adapted too well to train noise and loses quality on new data. In ML, almost everything revolves around finding a balance between these two extremes.

Regularization is needed to prevent the model from using too “aggressive” parameters or too complex rules. In linear models these are L1/L2 weight penalties. In trees, these are restrictions on the depth and size of leaves. In boosting this is shrinkage, the number of trees and early stopping.

An ML model cannot be fairly evaluated on the same data on which you selected the hyperparameters. Therefore, the data is divided into train, validation and test. When there is not enough data, they use cross-validation: they retrain the model several times on different folds and average the quality.

Regression is a task where the target is continuous: price, delivery time, sales volume, probability of default as a number from 0 to 1, temperature, lifetime value. The main question here is not “to which class to classify”, but “how large the output will be.”

Classification is tasks with discrete tags: spam / not spam, fraud / not fraud, product class, text language, disease. Often, the model first produces a probability or score, and only then the system turns it into a class through a threshold.

In many systems, the model does not produce a final solution, but a score. This score can be calibrated to probability, cut by a threshold, or used only for ranking. Conversations about precision/recall and business trade-off almost always revolve around this layer, and not around the “model architecture”.

In unsupervised learning there is no ready-made `y` that the model must guess. Instead, they look for structure in the data itself: groups of similar objects, low-dimensional representation, density, outliers. This is often not a “prediction” in the narrow sense, but a way to organize the feature space.

Linear models assume that the final response can be expressed as a weighted sum of features. They are not always the most accurate, but they are almost always needed as a baseline: they are fast, understandable, cheap, easy to debug, and show well whether there is at least a simple signal in the data.

Despite the name, logistic regression solves the classification problem. It builds a linear score, then passes it through the logistic function and gets the class probability. Therefore, this is one of the most standard and strong baselines for binary classification.

kNN teaches almost nothing in the classical sense: it stores train examples and, with a new request, looks for the nearest objects in the feature space. For classification it does voting, for regression it averages the target neighbors. This is a very intuitive, but sensitive to the scale of features and the size of the dataset.

Naive Bayes relies on the Bayes theorem and a rough but convenient assumption: features are conditionally independent given a fixed class. This assumption is often literally false, but the model can still perform surprisingly well, especially in text classification and as a very fast baseline.

Support Vector Machine builds a dividing surface between classes so that the distance to the nearest training examples is maximum. In the linear case, this is a neat geometric model of the boundary. With the kernel trick, SVM can also work with nonlinear partitions, although it scales worse than modern tabular ensembles.

Decision tree builds a tree of conditions like `if age < 30 and income > 100k`. At each node, it selects the partition that best reduces impurity or error. Trees are easy to read, they are able to catch nonlinearities and interactions of features, but alone they are unstable and prone to overfit.

Bagging reduces variance by averaging across many highly variable models. Random Forest is bagging over trees: each tree learns from a bootstrap subsample and a random subset of features. As a result, a forest is usually more stable and stronger than a single tree.

Boosting is not built as parallel voting, but as sequential error correction. Each new tree tries to improve where the current ensemble goes wrong. This is why gradient boosting and its successors often become the best models on tabular data with good feature preparation.

K-means looks for `K` centers and assigns each point to the nearest one. Then the centers are recalculated as the average of their points, and the cycle is repeated. This is a simple and very popular method of segmentation if the clusters are approximately “spherical” and the number of groups is known in advance.

DBSCAN does not require the number of clusters to be specified in advance. It looks at where the points are dense enough and considers such areas to be clusters. Objects that are not adjacent to any dense area are marked as noise. This is useful when clusters have complex shapes.

Principal Component Analysis looks for new axes in the data along which variation is greatest. Typically the first few components contain most of the "structure" of the data set. Therefore, PCA is used for visualization, dimensionality reduction, denoising, and feature preparation.

In classical ML, the quality of features is often more important than the choice of algorithm. Normalization, one-hot encoding, gap handling, time aggregations, logarithms, feature interactions, target encoding and domain features can give a greater gain than changing the model from logistic regression to something “smarter”.

Loss, which the model optimizes, and the metric by which the business makes a decision do not have to coincide. For regression, large errors may be important to you, and for classification, the prices of false positive and false negative are asymmetrical. Therefore, a conversation about quality always begins with the question: which error is more expensive and what exactly do we want to optimize.

One of the most common mistakes is to first convert the entire dataset and then divide it into train/test. This is how the model accidentally learns information from the future test. The correct pipeline ensures that any fit occurs only on the train fold, and then the same transform parameters are applied to validation/test.

An experienced ML process almost never starts with the most complex model. First, a simple baseline is built to understand the signal level in the data. Then they look at errors, fix symptoms and leakage, and only then tune the hyperparameters or move on to more powerful models.

Perceptron is a very simple linear classification unit: it sums features with weights, adds bias and applies threshold activation. In fact, this is already a model of the form `score = w x + b`, only with a nonlinear decision rule. Historically, this is one of the bridges between linear models and neural networks.

The key transition to neural networks did not occur because “different mathematics” suddenly appeared in them. The main idea remains the same: there is loss, there are parameters, we need to reduce the error. The novelty of backprop is that it can effectively distribute the error contribution back through many intermediate layers according to the chain rule.

Neural networks have changed, first of all, the power of the function and the way of working with features: now the model itself learns the representations, and does not depend so much on manual feature engineering. But the basic ML language has not gone away. There are still data, loss, train/val/test, overfitting, regularization, gradient descent and metrics.