Skip to main content
#linear-algebra#vector-norm#matrix-norm#regularization

Linear Algebra

Basic definitions and applications in machine learning, with a focus on vector and matrix norms.

A norm assigns a number to the magnitude of a vector or matrix. Norms are fundamental in linear algebra and recur in machine learning loss functions, regularization, and optimization constraints.

#1. What is a norm?

A norm measures the size of a vector or matrix. The most familiar example is the Euclidean norm, also called the β„“β‚‚ norm.

βˆ₯xβˆ₯2=βˆ‘i=1Nxi2\|\mathbf{x}\|_2 = \sqrt{\sum_{i=1}^{N} x_i^2}

Its square can be expressed as an inner product.

βˆ₯xβˆ₯22=x⊀x\|\mathbf{x}\|_2^2 = \mathbf{x}^{\top}\mathbf{x}

#2. Properties of a norm

A function f:RN→Rf : \mathbb{R}^N \to \mathbb{R} must satisfy the following four conditions to be a norm.

  1. Non-negativity
βˆ€x∈RN,f(x)β‰₯0\forall \mathbf{x} \in \mathbb{R}^N,\quad f(\mathbf{x}) \ge 0
  1. Definiteness
f(x)=0β€…β€ŠβŸΊβ€…β€Šx=0f(\mathbf{x}) = 0 \iff \mathbf{x} = \mathbf{0}
  1. Absolute homogeneity
βˆ€t∈R,f(tx)=∣t∣f(x)\forall t \in \mathbb{R},\quad f(t\mathbf{x}) = |t|f(\mathbf{x})
  1. Triangle inequality
f(x+y)≀f(x)+f(y)f(\mathbf{x}+\mathbf{y}) \le f(\mathbf{x}) + f(\mathbf{y})

#3. Common vector norms

#ℓ₁ norm (Manhattan norm)

βˆ₯xβˆ₯1=βˆ‘i=1N∣xi∣\|\mathbf{x}\|_1 = \sum_{i=1}^{N} |x_i|

Sum the absolute values of the components. ℓ₁ regularization can drive some coefficients exactly to zero, making it a common choice when sparsity is desirable.

#β„“βˆž norm (maximum norm)

βˆ₯xβˆ₯∞=max⁑i∣xi∣\|\mathbf{x}\|_{\infty} = \max_i |x_i|

Take the largest absolute component as the magnitude of the vector.

#The general β„“β‚š norm

βˆ₯xβˆ₯p=(βˆ‘i=1N∣xi∣p)1/p,pβ‰₯1\|\mathbf{x}\|_p = \left(\sum_{i=1}^{N} |x_i|^p\right)^{1/p},\qquad p \ge 1
  • At p = 1, this is the ℓ₁ norm.
  • At p = 2, this is the β„“β‚‚ norm.
  • As p approaches infinity, it converges to the β„“βˆž norm.
  • For 0 < p < 1, the expression fails the triangle inequality and therefore does not define a norm.

#4. A matrix norm: the Frobenius norm

Matrices also admit norms. One example is the Frobenius norm.

βˆ₯Aβˆ₯F=βˆ‘i=1Mβˆ‘j=1NAi,j2=tr⁑(A⊀A)\|\mathbf{A}\|_F = \sqrt{\sum_{i=1}^{M}\sum_{j=1}^{N} A_{i,j}^2} = \sqrt{\operatorname{tr}(\mathbf{A}^{\top}\mathbf{A})}

It equals the β„“β‚‚ norm of the long vector formed by stacking all matrix entries. Common uses include measuring the size of a matrix difference, low-rank approximation, and analyzing neural network weights.

#5. Uses in machine learning

  • ℓ₁ regularization encourages sparse parameters.
  • β„“β‚‚ regularization / weight decay penalizes large weights to reduce overfitting.
  • MSE and distance-based losses are closely related to β„“β‚‚ distance.
  • MAE is related to ℓ₁ distance.
  • Optimization constraints often bound the norm of a parameter vector or perturbation.

#Summary

Norms give a consistent definition of size for vectors and matrices. Choosing ℓ₁, β„“β‚‚, or β„“βˆž changes the geometry of an optimization problem and can change the resulting model. These tools recur in distance calculations, regularization, loss functions, and robustness analysis.

Connect