A norm assigns a number to the magnitude of a vector or matrix. Norms are fundamental in linear algebra and recur in machine learning loss functions, regularization, and optimization constraints.
#1. What is a norm?
A norm measures the size of a vector or matrix. The most familiar example is the Euclidean norm, also called the ββ norm.
Its square can be expressed as an inner product.
#2. Properties of a norm
A function must satisfy the following four conditions to be a norm.
- Non-negativity
- Definiteness
- Absolute homogeneity
- Triangle inequality
#3. Common vector norms
#ββ norm (Manhattan norm)
Sum the absolute values of the components. ββ regularization can drive some coefficients exactly to zero, making it a common choice when sparsity is desirable.
#ββ norm (maximum norm)
Take the largest absolute component as the magnitude of the vector.
#The general ββ norm
- At p = 1, this is the ββ norm.
- At p = 2, this is the ββ norm.
- As p approaches infinity, it converges to the ββ norm.
- For 0 < p < 1, the expression fails the triangle inequality and therefore does not define a norm.
#4. A matrix norm: the Frobenius norm
Matrices also admit norms. One example is the Frobenius norm.
It equals the ββ norm of the long vector formed by stacking all matrix entries. Common uses include measuring the size of a matrix difference, low-rank approximation, and analyzing neural network weights.
#5. Uses in machine learning
- ββ regularization encourages sparse parameters.
- ββ regularization / weight decay penalizes large weights to reduce overfitting.
- MSE and distance-based losses are closely related to ββ distance.
- MAE is related to ββ distance.
- Optimization constraints often bound the norm of a parameter vector or perturbation.
#Summary
Norms give a consistent definition of size for vectors and matrices. Choosing ββ, ββ, or ββ changes the geometry of an optimization problem and can change the resulting model. These tools recur in distance calculations, regularization, loss functions, and robustness analysis.