Completeness contract
This article follows all thirteen pages of 1_Ch2 Fundamentals of ML.pdf. Numbered and unnumbered displayed equations, meaningful inline mathematics, equations inside figures, question placeholders ??, captions, footnotes, references, and legible handwriting are individually linked to ledgers. Source blanks and suspected errors are preserved first; completed and corrected variants receive separate statuses.

Keep three layers distinct.
Separate the 2025 PDF reconstruction, editorial and mathematical verification, and the research update dated August 18, 2026. Later research does not retroactively rewrite the lecture’s claims.

#Ledger status

• PDF mapping: 13 of 13 pages
• Source equations: 103 (MAI-P2-001–MAI-P2-103)
• Equation statuses: source-exact, source-suspect, editorially-completed, corrected-variant
• Source PDF SHA-256: c6ac9d80c5ae2bc96f0aa8aec70c126f559e6dd5c36d51a46eb1bdb6775e836c
• Figures are reconstructed as new diagrams or precise descriptions preserving axes, legends, values, and structure, rather than reproducing source images.

#Reconstructing the PDF source

#2. Fundamentals of machine learning

Machine learning is introduced as methods that automatically discover patterns in large datasets and use them to predict future data or make decisions under uncertainty. The lecture presents it as a response to automated-analysis demands in the era of big data.

Even an apparently enormous dataset can contain few effective samples of a rare event of interest. The lecture describes this long-tail property: a few events are common, while most are rare.

#2.1 Tasks

#2.1.1 Classification

Classification learns a mapping from inputs to outputs. The output space is $y\in\{1,\ldots,C\}$. With $C=2$, the task is binary classification, usually using $y\in\{0,1\}$; with $C>2$, it is multiclass classification.

• Nonexclusive classes give multilabel classification, which can be viewed as a multioutput model predicting several related binary labels together.

Unless otherwise stated, classification here means single-output multiclass classification.

As function approximation, classification assumes an unknown relationship $y=f(x)$ and estimates it from labeled training examples.

After learning, $\hat y=\hat f(x)$ predicts new inputs. Working well on previously unseen inputs is generalization.

#2.1.2 Regression

Regression predicts numerical values from inputs. The learned function has form $f:\mathbb{R}^{N}\to\mathbb{R}$. Its structure resembles classification, but its outputs are real-valued.

The lecture compares a line and quadratic polynomial using one real input $x_l\in\mathbb{R}$ and response $y_l\in\mathbb{R}$. The page ends by asking which fit looks better, leaving ?? rather than supplying an answer.

Lecture annotation — PDF p. 1
A note appearing to read “path” and a λ symbol occur near the regression example. Context is insufficient to determine their meaning, so the ledger marks them uncertain to read.

#2.1.3 Density estimation

Density estimation learns $p_{\mathrm{model}}:\mathbb{R}^{N}\to\mathbb{R}$, interpreting $p_{\mathrm{model}}(x)$ as a PDF or PMF over the space generating samples.

A good density model learns data structure, including dense regions and rarely observed ones. Knowing a density does not automatically solve every related task: required computations can still be intractable.

#2.2 Example: linear regression

The input is $x\in\mathbb{R}^{N}$ and output is scalar $y\in\mathbb{R}$. Predictions are defined as follows.

Here, $w\in\mathbb{R}^{N}$ is the parameter vector. Each component multiplies its corresponding feature, determining that feature’s contribution to prediction.

The learning algorithm adjusts weights using training data $\left(X^{(\mathrm{train})},y^{(\mathrm{train})}\right)$. An intuitive criterion minimizes training mean squared error (MSE).

The lecture derives the normal equation by setting the gradient to zero without skipping intermediate lines.

The final expression is source equation (1). Its inverse form implicitly assumes an invertible Gram matrix.

The source attributes this derivative identity to The Matrix Cookbook.

Lecture annotation — PDF p. 2, training-sample notation
The note reads (x_l,y_l) as one training sample and stacks rowwise feature vectors to form X^(train).

Uncertain reading — lower-right PDF p. 2
A dimensional comparison resembling L ≫ N and a numerical example are obscured by overlapping handwriting. Their location is retained without speculative reconstruction.

#2.3 Underfitting and overfitting

The central machine-learning challenge is maintaining performance on inputs not used in training. This is generalization.

Optimizing parameters to reduce training-set error is an optimization problem. Machine learning adds the requirement of low generalization or test error: expected error on new inputs.

The linear-regression example evaluates the model from equation (1) on a test set.

Here, M is the number of test samples.

Training and test data are modeled as draws from a data-generating distribution. The typical i.i.d. assumption makes samples independent and gives training and test sets the same distribution.

For a procedure that samples training data, optimizes parameters, and then samples test data, the lecture explains that expected test error is no smaller than expected training error.

• A good learning algorithm should achieve low training error.

• It should also keep the gap between training and test errors small.

Underfitting occurs when the model cannot obtain sufficiently low error even on training data.

Overfitting occurs when the training–test error gap is excessively large.

Changing model capacity controls these possibilities. Low capacity may fail to represent the training set; high capacity may memorize properties that do not help on test data.

#2.3.1 No free lunch theorem

“All models are wrong, but some are useful.” — George Box

Machine learning designs models and fitting algorithms. Cross-validation can select empirically better methods for a particular problem, but no single model is optimal for every problem: assumptions useful in one domain can hurt in another.

Uncertain reading — bottom of PDF p. 3
Handwriting following “example” cannot be determined reliably because of image resolution and overlapping strokes.

The no-free-lunch result motivates multiple model families for varied real data structures. Even the same model has different speed, accuracy, and complexity trade-offs under different training algorithms.

#2.3.2 Regularization

A learning algorithm can incorporate preferences suited to a task. Performance improves when those preferences match the problem’s structure.

The linear-regression example adds weight decay to favor smaller weights.

λ is selected in advance. With $\lambda=0$ there is no small-weight preference; larger λ shrinks weights more strongly.

Figure 3 compares underfitting at excessive λ, a suitable intermediate fit, and overfitting near zero.

Preferences can be explicit or implicit. This example explicitly favors small-weight linear functions through an added objective term.

Lecture annotation — right of Figure 2, PDF p. 4
Added error-versus-iteration curves distinguish three trends appearing to indicate training, test, and underfitting.

Uncertain reading — lower-right PDF p. 4
Partially legible short notes are not completed by guessing their meaning.

These different ways of introducing preferences are collectively called regularization.

Regularization is any modification of a learning algorithm intended to reduce generalization error rather than training error.

Just as no algorithm is best for every problem, no regularizer is universally best. Choose one suited to the task.

The lecture summarizes deep learning’s philosophy as solving broad classes of tasks effectively through very general forms of regularization.

#2.4 Hyperparameters and validation sets

Most learning algorithms have hyperparameters controlling their behavior. These are not the model parameters adapted by the algorithm itself.

Polynomial degree in Figure 1 is a capacity hyperparameter, as is weight-decay λ.

Some settings remain hyperparameters because optimization is difficult. More commonly, learning them on the training set is inappropriate: choosing capacity there can favor the maximum available capacity and overfit.

A validation set unseen by the training algorithm addresses this problem.

#2.4.1 Training, validation, and test sets

• Training set: learns model parameters.

• Validation set: estimates generalization to unseen data and selects hyperparameters and configurations.

• Test set: evaluates the final frozen model on genuinely unseen data for an unbiased generalization estimate.

Lecture annotation — PDF p. 5
Notes appearing to read “hyper” beside the hyperparameter heading and “dropout” near regularization are present.

#2.4.2 Cross-validation

With a small test set in a fixed split, statistical uncertainty in mean test error makes claims that algorithm A beats B difficult.

This matters less with hundreds of thousands of samples. Small datasets can instead spend extra computation on repeated training and evaluation across splits, using all samples to estimate mean test error.

The usual K-fold procedure divides data into K disjoint subsets and runs K experiments. Run k tests on subset k and trains on the remainder; average the K test errors.

#2.5 Supervised-learning example: support vector machines

Like logistic regression, an SVM centers on a linear function $w^{\mathsf T}x+b$ but outputs class identity rather than probability. It predicts positive for $w^{\mathsf T}x+b>0$ and negative for $w^{\mathsf T}x+b<0$.

Its key innovation is the kernel trick. Learning algorithms expressible through sample inner products can fit linear models in transformed spaces without explicitly constructing high-dimensional features.

The footnote introduces logistic regression as generalized linear regression for binary classification: targets are $y\in\{0,1\}$, and $\sigma(w^{\mathsf T}x+b)$ squashes the linear output for probabilistic interpretation.

The SVM linear function can be rewritten as a sum of inner products with training samples.

Here, x_l is a training sample and α a coefficient vector. Map inputs through φ(x), then replace inner products with a kernel.

The function may be nonlinear in x while remaining linear in φ(x) and in α. This corresponds exactly to transforming every input through φ and learning a linear model in that space.

• With φ fixed and only α optimized, a model nonlinear in original inputs can be learned through convex optimization.

• κ may be much cheaper to compute than explicitly constructing and dotting the high-dimensional feature vectors.

The lecture’s representative example is the Gaussian kernel.

The notation $\mathcal{N}(x;\mu,\Sigma)$ denotes a normal density with mean μ and covariance Σ.

#2.6 Minimum-distance classifiers: deterministic classification

A minimum-distance classifier calculates distance from a pattern vector to each class prototype and assigns the nearest prototype’s class.

Prototypes are usually class-mean vectors.

M_c counts patterns used for mean c, S_c is pattern class c, and C is the number of classes.

Using Euclidean similarity gives the following distances.

Assign x to the class whose distance is smaller than every other class’s.

Ties with $D_c(x)=D_{c\prime}(x)$ are resolved arbitrarily.

Minimizing distance is equivalent to maximizing the following discriminant.