Skip to main content
Park JiHo Research notes · Seoul
KO

· Translation updated

Modern Artificial Intelligence I — From AI, ML, and DL to Probability and Optimization

Integrated ECE5992 Chapter 1 study notes covering AI, ML, deep learning, image processing, linear algebra, perceptrons, convolution, random variables, Gaussian distributions, WSS, losses, and gradient descent, preserving the equations from both lecture handouts.

한국어 원문

Editorial and copyright principles
This English edition translates a Korean study-oriented reconstruction of ECE5992 Modern Artificial Intelligence Chapter 1 materials dated March 12 and 19, 2025. It does not reproduce the source’s sentence arrangement, page layout, or illustrations. Concepts are explained in new wording and diagrams are newly created. The mathematical content is preserved because changing an equation’s meaning would be unacceptable. Differences in equation numbering between versions and questionable source notation are documented separately.

No material was discarded in this consolidation.
Earlier provisional drafts on linear algebra, perceptrons, probability, and optimization, temporarily numbered 2–8, were merged into this first article. This integrated edition combines two Chapter 1 PDFs in one post.

#Preserving every equation

Each displayed equation has a unique MAI identifier. The integrated article currently contains 238 equation blocks. Identical equations repeated across source versions are explained once, while differences in notation or numbering receive notes. KaTeX rendering rather than equation images preserves copying, search, and accessibility.

#Series structure

The series follows ECE5992 materials actually obtained: this integrated Chapter 1 plus seven subsequent PDFs form eight articles. No empty posts are created merely to fill numbers.

The eight-part study map above presents this progression in one view.

Mathematical and probability foundations → Machine-learning generalization → Perceptrons and CNNs → Gradient optimization → Image classification → Semantic segmentation → VAEs and diffusion → Contrastive representation learning

Map entries become links as subsequent articles become available, so a duplicate table organized around PDF filenames is unnecessary here.

#Auditing source notation

Comparison of the two versions distinguishes source notation from standard mathematical notation in the following cases.

  1. The second equation in the two-variable linear-system example contains x_2 twice. The source equation is preserved and a general form is given separately.
  2. If A\in\mathbb{R}^{M\times N} and x\in\mathbb{R}^{N}, then Ax\in\mathbb{R}^{M}. A source sentence’s output dimension appears to be a typographical error in context; the text uses the correct dimension.
  3. The geometric distance from the origin to a hyperplane is |b|/\|w\|_2, while b/\|w\|_2 can represent signed distance. The source’s b/\|w\|_2 is annotated as a signed expression.
  4. The joint density of M independent Gaussian vectors requires normalization |C|^{-M/2}. A source derivation appears to use |C|^{-1/2}, so both the preserved source expression and its correction are shown.
  5. For complex random vectors, uncorrelatedness generally requires E[\widetilde X_i\widetilde X_j^*]=0. One source version omits conjugation, and the difference is identified in the text.

#Part 1.0: Fields and the research ecosystem

Equation reading mode — Every displayed equation below opens Explanation and calculations. Each card proceeds through the question → intermediate calculations → numerical example → checks → external references.

#2. AI, ML, and DL are different terms

Broadly, artificial intelligence concerns building systems that behave intelligently. Machine learning learns rules or representations from data within that field. Deep learning is a machine-learning family centered on multilayer neural networks.

In one line:

Artificial intelligence ⊃ Machine learning ⊃ Deep learning

This inclusion diagram is not a complete law of the field’s history. Rule-based AI, search, planning, and optimization can be non-learning approaches, while modern systems combine learned models with search, tools, external memory, and planners. It provides a starting view of field relationships rather than a complete account of contemporary architectures.

#3. What should you read?

The research ecosystem presented in the lectures remains useful. Following a field requires more than reading isolated papers: develop a habit of scanning journal and conference proceedings tables of contents.

#Representative journals

  • IEEE Transactions on Pattern Analysis and Machine Intelligence
  • Nature Machine Intelligence
  • IEEE Transactions on Image Processing
  • IEEE Transactions on Signal Processing
  • IEEE Transactions on Medical Imaging
  • IEEE Transactions on Computational Imaging
  • IEEE Transactions on Intelligent Transportation Systems
  • IEEE Transactions on Robotics
  • Pattern Recognition
  • Medical Image Analysis
  • IEEE Signal Processing Letters
  • IEEE Robotics and Automation Letters

#Representative conferences

  • NeurIPS
  • ICML
  • ICLR
  • CVPR
  • ICCV
  • ECCV
  • MICCAI
  • ICIP
  • IROS
  • ICRA

The important point is connections between fields, rather than the list alone. Imaging, signal processing, robotics, medical imaging, and machine learning overlap extensively in actual research.

#4. Image processing and computer vision

The lecture presents the imaging pipeline as follows.

Physical scene → Image formation → Raw measurements → Image processing → Improved image → Image analysis → Meaningful decisions

Two additional branches accompany this pipeline.

  • Image display: representing images for human viewing
  • Image compression: reducing storage and transmission costs

Traditionally, image processing was often called low-level vision, and image analysis high-level vision. Deep learning has weakened this boundary: one network can jointly learn restoration, feature extraction, detection, and classification.

#Low-level and high-level perspectives

PerspectiveLow-level emphasisHigh-level emphasis
Main questionHow should pixels be restored or transformed?What is in the scene, and what does it mean?
Typical tasksDenoising, super-resolution, restoration, calibrationClassification, detection, segmentation, tracking, reasoning
Traditional foundationsSignal processing, optimization, statisticsPattern recognition, machine learning
Modern characteristicSubstantial overlapSubstantial overlap

#5. An important caution from the lectures

One central message is that fundamental understanding of modern deep learning remains immature.

Better model performance differs from fully understanding why models generalize, what representations they learn, and when they fail. This motivates the linear algebra, probability, and optimization that follow.

#6. August 2026 update

These foundations remain necessary in 2026, while implementation practice has expanded substantially.

  • The source article identifies the current stable PyTorch documentation as the 2.13 series, with torch.linalg, torch.func, torch.compile, distributed training, and attention kernels alongside tensor operations in one ecosystem.
  • CNNs are no longer the only basic visual architecture. Since Vision Transformer, patch-based attention has become a standard choice, with CNNs, Transformers, and hybrids coexisting.
  • “Modern AI equals a simple multilayer network” is therefore incomplete. Connect mathematical foundations → representation learning → large-scale optimization → efficient compilation and distributed execution.

#Current references in the source article

#7. Next section

Next comes linear algebra, beginning with vector and matrix notation and reconstructing inner products, outer products, matrix–vector products, and matrix–matrix products equation by equation.

#Part 1.1: Linear Algebra I — Vectors, Matrices, and Products

#1. Why begin with linear algebra?

Most modern neural networks ultimately execute operations on vectors and matrices. Linear algebra is more than a prerequisite: it connects forward passes, matrix representations of convolution, attention, optimization, and covariance calculations.

#1.1 A two-variable linear system

The lectures provide the following two equations. The second contains x_2 twice in the source. We preserve that notation and give a conventional two-variable system immediately below.

Part I · Study equation

Identify the equation's role and symbols
A1,1x1−A1,2x2=b1,A2,1x2−A2,2x2=b2.\begin{aligned} A_{1,1}x_1 - A_{1,2}x_2 &= b_1,\\ A_{2,1}x_2 - A_{2,2}x_2 &= b_2. \end{aligned}
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-001 Identifier: MAI2-001

A general two-variable linear system is usually written as follows.

Part I · Study equation

Identify the equation's role and symbols
A1,1x1+A1,2x2=b1,A2,1x1+A2,2x2=b2.\begin{aligned} A_{1,1}x_1 + A_{1,2}x_2 &= b_1,\\ A_{2,1}x_1 + A_{2,2}x_2 &= b_2. \end{aligned}
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-002 Identifier: MAI2-002

Matrix notation makes it much more compact.

Part I · Study equation

Matrix multiplication repeats row–column inner products
Ax=b,A=[A1,1A1,2A2,1A2,2],b=[b1b2].A\mathbf{x}=\mathbf{b},\qquad A=\begin{bmatrix}A_{1,1}&A_{1,2}\\A_{2,1}&A_{2,2}\end{bmatrix},\qquad \mathbf{b}=\begin{bmatrix}b_1\\b_2\end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-003 Identifier: MAI2-003

#2. Basic notation

Always check column-vector and matrix dimensions first.

Part I · Study equation

Identify the equation's role and symbols
x∈RM,xT is a row vector.\mathbf{x}\in\mathbb{R}^{M},\qquad \mathbf{x}^{T}\ \text{is a row vector}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-004 Identifier: MAI2-004

Part I · Study equation

Identify the equation's role and symbols
A∈RM×N.A\in\mathbb{R}^{M\times N}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-005 Identifier: MAI2-005

Part I · Study equation

Matrix multiplication repeats row–column inner products
x=[x1x2⋮xM].\mathbf{x}= \begin{bmatrix} x_1\\x_2\\\vdots\\x_M \end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-006 Identifier: MAI2-006

Part I · Study equation

Matrix multiplication repeats row–column inner products
A=[A1,1A1,2⋯A1,NA2,1A2,2⋯A2,N⋮⋮⋱⋮AM,1AM,2⋯AM,N].A= \begin{bmatrix} A_{1,1}&A_{1,2}&\cdots&A_{1,N}\\ A_{2,1}&A_{2,2}&\cdots&A_{2,N}\\ \vdots&\vdots&\ddots&\vdots\\ A_{M,1}&A_{M,2}&\cdots&A_{M,N} \end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-007 Identifier: MAI2-007

Columnwise and rowwise notation are also common.

Part I · Study equation

Matrix multiplication repeats row–column inner products
A=[a1a2⋯aN].A=\begin{bmatrix}\mathbf{a}_1&\mathbf{a}_2&\cdots&\mathbf{a}_N\end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-008 Identifier: MAI2-008

Part I · Study equation

Matrix multiplication repeats row–column inner products
A=[a1Ta2T⋮aMT].A=\begin{bmatrix} \mathbf{a}_1^T\\ \mathbf{a}_2^T\\ \vdots\\ \mathbf{a}_M^T \end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-009 Identifier: MAI2-009

#3. Vectorizing an image

Flattening a multidimensional image into one long vector allows operations such as convolution to be expressed as matrix–vector multiplication.

Part I · Study equation

Matrix multiplication repeats row–column inner products
[y1⋯yN]⟶[y1⋮yN].\begin{bmatrix}y_1&\cdots&y_N\end{bmatrix} \longrightarrow \begin{bmatrix}y_1\\\vdots\\y_N\end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-010 Identifier: MAI2-010

Part I · Study equation

Identify the equation's role and symbols
y[m,n]=(a∗x)[m,n].y[m,n]=(a*x)[m,n].
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-011 Identifier: MAI2-011

Part I · Study equation

Solving simultaneous equations means satisfying all constraints together
y=Ax.\mathbf{y}=A\mathbf{x}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which unknown vector makes every row's equation hold simultaneously?

Key idea

Read each row as an equation and solve using elimination, an inverse, or a factorization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Expand into rowwise equations

    One matrix row dotted with the unknown vector produces one right-hand-side value.

    Ai,:x=biA_{i,:}x=b_i
  2. 2

    Eliminate a variable

    Add or subtract equations to reduce the unknown count. LU or QR factorizations play this role in larger problems.

  3. 3

    Back-substitute the remaining value

    Insert the solved value into an original equation to obtain other unknowns.

  4. 4

    Substitute into every equation again

    Check that the calculated solution satisfies every row.

    Ax=?bAx\stackrel{?}=b

Worked example

Work through the numbers — Solving a 2×2 system directly

Think of finding the intersection of two lines.

  1. 1

    Problem

    2x+y=5,x−y=12x+y=5,\qquad x-y=1
  2. 2

    Add equations to eliminate y

    3x=6⇒x=23x=6\Rightarrow x=2
  3. 3

    Back-substitute into the second equation

    2−y=1⇒y=12-y=1\Rightarrow y=1
  4. 4

    Check

    2(2)+1=5,2−1=12(2)+1=5,\qquad 2-1=1

Result: The solution is therefore (x,y)=(2,1).

Sanity check

Checks
  • Ax on the left and b on the right must have the same dimension.
  • The inverse expression applies directly only when A is square and invertible.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-012 Identifier: MAI2-012

#4. Vector–vector products

#4.1 Inner product

An inner product reduces two vectors to one scalar.

Part I · Study equation

Matrix multiplication repeats row–column inner products
xTy=[x1x2⋯xN][y1y2⋮yN]=∑i=1Nxiyi∈R.\mathbf{x}^T\mathbf{y} = \begin{bmatrix}x_1&x_2&\cdots&x_N\end{bmatrix} \begin{bmatrix}y_1\\y_2\\\vdots\\y_N\end{bmatrix} = \sum_{i=1}^{N}x_i y_i\in\mathbb{R}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Formula-specific lesson

Connect dot products, outer products, matrix products, norms, rank, and inverses in one numerical example

Use the same x, y, A, and B to calculate dot and outer products, Ax, AB, outer-sum decomposition, norms, determinants, rank, and inverses, connecting Part I's key linear-algebra identities.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-013 Identifier: MAI2-013

Part I · Study equation

Identify the equation's role and symbols
xTy=yTx.\mathbf{x}^T\mathbf{y}=\mathbf{y}^T\mathbf{x}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-014 Identifier: MAI2-014

#4.2 Outer product

An outer product forms a matrix from two vectors.

Part I · Study equation

Matrix multiplication repeats row–column inner products
xyT=[x1y1x1y2⋯x1yNx2y1x2y2⋯x2yN⋮⋮⋱⋮xMy1xMy2⋯xMyN]∈RM×N.\mathbf{x}\mathbf{y}^T = \begin{bmatrix} x_1y_1&x_1y_2&\cdots&x_1y_N\\ x_2y_1&x_2y_2&\cdots&x_2y_N\\ \vdots&\vdots&\ddots&\vdots\\ x_My_1&x_My_2&\cdots&x_My_N \end{bmatrix} \in\mathbb{R}^{M\times N}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Formula-specific lesson

Connect dot products, outer products, matrix products, norms, rank, and inverses in one numerical example

Use the same x, y, A, and B to calculate dot and outer products, Ax, AB, outer-sum decomposition, norms, determinants, rank, and inverses, connecting Part I's key linear-algebra identities.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-015 Identifier: MAI2-015

#5. Two views of matrix–vector multiplication

Rowwise, each output component is the inner product of one matrix row with the input vector.

Part I · Study equation

Matrix multiplication repeats row–column inner products
y=Ax=[a1Txa2Tx⋮aMTx].\mathbf{y}=A\mathbf{x} = \begin{bmatrix} \mathbf{a}_1^T\mathbf{x}\\ \mathbf{a}_2^T\mathbf{x}\\ \vdots\\ \mathbf{a}_M^T\mathbf{x} \end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Formula-specific lesson

Connect dot products, outer products, matrix products, norms, rank, and inverses in one numerical example

Use the same x, y, A, and B to calculate dot and outer products, Ax, AB, outer-sum decomposition, norms, determinants, rank, and inverses, connecting Part I's key linear-algebra identities.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-016 Identifier: MAI2-016

Part I · Study equation

Identify the equation's role and symbols
yi=aiTx.y_i=\mathbf{a}_i^T\mathbf{x}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-017 Identifier: MAI2-017

Columnwise, the output is a linear combination of A’s columns.

Part I · Study equation

Matrix multiplication repeats row–column inner products
y=Ax=[a1a2⋯aN][x1x2⋮xN]=a1x1+a2x2+⋯+aNxN.\mathbf{y}=A\mathbf{x} = \begin{bmatrix}\mathbf{a}_1&\mathbf{a}_2&\cdots&\mathbf{a}_N\end{bmatrix} \begin{bmatrix}x_1\\x_2\\\vdots\\x_N\end{bmatrix} = \mathbf{a}_1x_1+\mathbf{a}_2x_2+\cdots+\mathbf{a}_Nx_N.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-018 Identifier: MAI2-018

A row vector can also multiply from the left.

Part I · Study equation

Matrix multiplication repeats row–column inner products
yT=xTA=[xTa1xTa2⋯xTaN].\mathbf{y}^T=\mathbf{x}^T A = \begin{bmatrix} \mathbf{x}^T\mathbf{a}_1& \mathbf{x}^T\mathbf{a}_2& \cdots& \mathbf{x}^T\mathbf{a}_N \end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-019 Identifier: MAI2-019

Part I · Study equation

Identify the equation's role and symbols
yT=xTA=x1a1T+x2a2T+⋯+xMaMT.\mathbf{y}^T=\mathbf{x}^T A = x_1\mathbf{a}_1^T+x_2\mathbf{a}_2^T+\cdots+x_M\mathbf{a}_M^T.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-020 Identifier: MAI2-020

#6. Matrix–matrix multiplication

Part I · Study equation

Identify the equation's role and symbols
A∈RM×N,B∈RN×P,C=AB∈RM×P.A\in\mathbb{R}^{M\times N},\quad B\in\mathbb{R}^{N\times P},\quad C=AB\in\mathbb{R}^{M\times P}.
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

Connect dot products, outer products, matrix products, norms, rank, and inverses in one numerical example

Use the same x, y, A, and B to calculate dot and outer products, Ax, AB, outer-sum decomposition, norms, determinants, rank, and inverses, connecting Part I's key linear-algebra identities.

Structure explanation
Loading the lesson matched to this equation and derivation step.

Symbols

Symbols used in this equation
SymbolMeaning
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-021 Identifier: MAI2-021

Part I · Study equation

An inner product compresses matching component products into one score
Ci,j=∑k=1NAi,kBk,j.C_{i,j}=\sum_{k=1}^{N}A_{i,k}B_{k,j}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How much direction do two vectors share?

Key idea

Summing component products gives one scalar, repeatedly used for prediction scores, similarity, and projection.

Formula-specific lesson

Connect dot products, outer products, matrix products, norms, rank, and inverses in one numerical example

Use the same x, y, A, and B to calculate dot and outer products, Ax, AB, outer-sum decomposition, norms, determinants, rank, and inverses, connecting Part I's key linear-algebra identities.

Synchronized derivation
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Match the vector lengths

    They must have the same number of components.

  2. 2

    Multiply corresponding components

    x1y1, x2y2,…,xNyNx_1y_1,\ x_2y_2,\ldots,x_Ny_N
  3. 3

    Sum all products

    xTy=∑n=1Nxnynx^Ty=\sum_{n=1}^{N}x_ny_n
  4. 4

    Interpret sign and magnitude

    Positive generally indicates alignment, zero orthogonality, and negative opposition.

Worked example

Work through the numbers — Computing a weighted sum

Apply weights to two features.

  1. 1

    Values

    w=[2,−1]T,x=[3,4]Tw=[2,-1]^T,\quad x=[3,4]^T
  2. 2

    Componentwise products

    [2⋅3, (−1)⋅4]=[6,−4][2\cdot3,\ (-1)\cdot4]=[6,-4]
  3. 3

    Sum

    wTx=6−4=2w^Tx=6-4=2

Result: The final linear score is 2.

Sanity check

Checks
  • The result is one scalar, not a vector.
  • For quantities with units, product terms must share units to be added.

Symbols

Symbols used in this equation
SymbolMeaning
∑\sumSum over all specified terms
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-022 Identifier: MAI2-022

Collecting row–column inner products gives:

Part I · Study equation

Matrix multiplication repeats row–column inner products
C=AB=[a1Tb1a1Tb2⋯a1TbPa2Tb1a2Tb2⋯a2TbP⋮⋮⋱⋮aMTb1aMTb2⋯aMTbP].C=AB= \begin{bmatrix} \mathbf{a}_1^T\mathbf{b}_1&\mathbf{a}_1^T\mathbf{b}_2&\cdots&\mathbf{a}_1^T\mathbf{b}_P\\ \mathbf{a}_2^T\mathbf{b}_1&\mathbf{a}_2^T\mathbf{b}_2&\cdots&\mathbf{a}_2^T\mathbf{b}_P\\ \vdots&\vdots&\ddots&\vdots\\ \mathbf{a}_M^T\mathbf{b}_1&\mathbf{a}_M^T\mathbf{b}_2&\cdots&\mathbf{a}_M^T\mathbf{b}_P \end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Symbols

Symbols used in this equation
SymbolMeaning
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-023 Identifier: MAI2-023

The column–row view expresses it as a sum of outer products.

Part I · Study equation

An inner product compresses matching component products into one score
C=AB=∑i=1NaibiT.C=AB=\sum_{i=1}^{N}\mathbf{a}_i\mathbf{b}_i^T.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How much direction do two vectors share?

Key idea

Summing component products gives one scalar, repeatedly used for prediction scores, similarity, and projection.

Formula-specific lesson

Connect dot products, outer products, matrix products, norms, rank, and inverses in one numerical example

Use the same x, y, A, and B to calculate dot and outer products, Ax, AB, outer-sum decomposition, norms, determinants, rank, and inverses, connecting Part I's key linear-algebra identities.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Match the vector lengths

    They must have the same number of components.

  2. 2

    Multiply corresponding components

    x1y1, x2y2,…,xNyNx_1y_1,\ x_2y_2,\ldots,x_Ny_N
  3. 3

    Sum all products

    xTy=∑n=1Nxnynx^Ty=\sum_{n=1}^{N}x_ny_n
  4. 4

    Interpret sign and magnitude

    Positive generally indicates alignment, zero orthogonality, and negative opposition.

Worked example

Work through the numbers — Computing a weighted sum

Apply weights to two features.

  1. 1

    Values

    w=[2,−1]T,x=[3,4]Tw=[2,-1]^T,\quad x=[3,4]^T
  2. 2

    Componentwise products

    [2⋅3, (−1)⋅4]=[6,−4][2\cdot3,\ (-1)\cdot4]=[6,-4]
  3. 3

    Sum

    wTx=6−4=2w^Tx=6-4=2

Result: The final linear score is 2.

Sanity check

Checks
  • The result is one scalar, not a vector.
  • For quantities with units, product terms must share units to be added.

Symbols

Symbols used in this equation
SymbolMeaning
∑\sumSum over all specified terms
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-024 Identifier: MAI2-024

Viewing B by columns, each column of C is A multiplied by one column of B.

Part I · Study equation

Matrix multiplication repeats row–column inner products
C=A[b1b2⋯bP]=[Ab1Ab2⋯AbP].C=A\begin{bmatrix}\mathbf{b}_1&\mathbf{b}_2&\cdots&\mathbf{b}_P\end{bmatrix} = \begin{bmatrix}A\mathbf{b}_1&A\mathbf{b}_2&\cdots&A\mathbf{b}_P\end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Symbols

Symbols used in this equation
SymbolMeaning
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-025 Identifier: MAI2-025

Part I · Study equation

Identify the equation's role and symbols
ci=Abi.\mathbf{c}_i=A\mathbf{b}_i.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-026 Identifier: MAI2-026

Viewing A by rows, each row of C is obtained as follows.

Part I · Study equation

Matrix multiplication repeats row–column inner products
C=[a1TBa2TB⋮aMTB].C= \begin{bmatrix} \mathbf{a}_1^TB\\ \mathbf{a}_2^TB\\ \vdots\\ \mathbf{a}_M^TB \end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Symbols

Symbols used in this equation
SymbolMeaning
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI2-027 Identifier: MAI2-027

Part I · Study equation

Identify the equation's role and symbols
ciT=aiTB.\mathbf{c}_i^T=\mathbf{a}_i^TB.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-028 Identifier: MAI2-028

#6.1 Basic multiplication properties

Part I · Study equation

Identify the equation's role and symbols
(AB)C=A(BC).(AB)C=A(BC).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-029 Identifier: MAI2-029

Part I · Study equation

Identify the equation's role and symbols
A(B+C)=AB+AC.A(B+C)=AB+AC.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-030 Identifier: MAI2-030

Part I · Study equation

Identify the equation's role and symbols
AB≠BAin general.AB\neq BA\quad\text{in general}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-031 Identifier: MAI2-031

#7. Identity and diagonal matrices

Part I · Study equation

Identify the equation's role and symbols
Ii,j={1,i=j,0,i≠j.I_{i,j}= \begin{cases} 1,&i=j,\\ 0,&i\neq j. \end{cases}
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-032 Identifier: MAI2-032

Part I · Study equation

Identify the equation's role and symbols
AI=A=IA.AI=A=IA.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-033 Identifier: MAI2-033

Part I · Study equation

Identify the equation's role and symbols
D=diag⁡(d1,…,dN),Di,j={di,i=j,0,i≠j.D=\operatorname{diag}(d_1,\ldots,d_N),\qquad D_{i,j}= \begin{cases} d_i,&i=j,\\ 0,&i\neq j. \end{cases}
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-034 Identifier: MAI2-034

Part I · Study equation

Identify the equation's role and symbols
I=diag⁡(1,…,1).I=\operatorname{diag}(1,\ldots,1).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-035 Identifier: MAI2-035

#8. Transpose

Part I · Study equation

Identify the equation's role and symbols
(AT)i,j=Aj,i.(A^T)_{i,j}=A_{j,i}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-036 Identifier: MAI2-036

Part I · Study equation

Identify the equation's role and symbols
(AT)T=A.(A^T)^T=A.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-037 Identifier: MAI2-037

Part I · Study equation

Identify the equation's role and symbols
(AB)T=BTAT.(AB)^T=B^TA^T.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-038 Identifier: MAI2-038

Part I · Study equation

Identify the equation's role and symbols
(A+B)T=AT+BT.(A+B)^T=A^T+B^T.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI2-039 Identifier: MAI2-039

#9. Implementation perspective in 2026

As of the source’s August 2026 update, PyTorch 2.13’s torch.linalg directly provides nearly all operations covered here, including norms, determinants, rank, solve, QR, and Cholesky.

In practice, solving the linear system Ax=b with torch.linalg.solve is often numerically and computationally preferable to explicitly forming A^{-1}b.

#10. Perspectives to remember

Rather than memorize multiplication as one formula, learn all four views.

  1. Row–column inner products
  2. A sum of outer products
  3. Applying A to each column of B
  4. Applying B to each row of A

These perspectives recur in fully connected layers, convolution’s im2col representation, and attention projections.

#Part 1.2: Linear Algebra II — Symmetry, Trace, Norms, Rank, and Determinants

#1. Symmetric and skew-symmetric matrices

Part I · Study equation

Identify the equation's role and symbols
A=AT.A=A^T.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-001 Identifier: MAI3-001

Part I · Study equation

Identify the equation's role and symbols
A=−AT.A=-A^T.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-002 Identifier: MAI3-002

Any square matrix decomposes into symmetric and skew-symmetric parts.

Part I · Study equation

Identify the equation's role and symbols
A=12(A+AT)+12(A−AT).A= \frac12(A+A^T)+\frac12(A-A^T).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-003 Identifier: MAI3-003

#2. Trace

Trace is the sum of diagonal entries.

Part I · Study equation

Identify the equation's role and symbols
tr⁡(A)=∑i=1NAi,i.\operatorname{tr}(A)=\sum_{i=1}^{N}A_{i,i}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-004 Identifier: MAI3-004

Part I · Study equation

Identify the equation's role and symbols
tr⁡(A)=tr⁡(AT).\operatorname{tr}(A)=\operatorname{tr}(A^T).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-005 Identifier: MAI3-005

Part I · Study equation

Identify the equation's role and symbols
tr⁡(A+B)=tr⁡(A)+tr⁡(B).\operatorname{tr}(A+B)=\operatorname{tr}(A)+\operatorname{tr}(B).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-006 Identifier: MAI3-006

Part I · Study equation

Identify the equation's role and symbols
tr⁡(tA)=t tr⁡(A).\operatorname{tr}(tA)=t\,\operatorname{tr}(A).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-007 Identifier: MAI3-007

Part I · Study equation

Identify the equation's role and symbols
tr⁡(AB)=tr⁡(BA).\operatorname{tr}(AB)=\operatorname{tr}(BA).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-008 Identifier: MAI3-008

Part I · Study equation

Identify the equation's role and symbols
tr⁡(ABC)=tr⁡(BCA)=tr⁡(CAB).\operatorname{tr}(ABC)=\operatorname{tr}(BCA)=\operatorname{tr}(CAB).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-009 Identifier: MAI3-009

#3. Norms

#3.1 Euclidean norm

Part I · Study equation

A norm summarizes many components as one magnitude
∥x∥2=∑i=1Nxi2.\|\mathbf{x}\|_2=\sqrt{\sum_{i=1}^{N}x_i^2}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How far is a vector or error from the origin?

Key idea

The L2 norm sums squared components and takes the square root. Squared norm omits the final square root.

Formula-specific lesson

Connect dot products, outer products, matrix products, norms, rank, and inverses in one numerical example

Use the same x, y, A, and B to calculate dot and outer products, Ax, AB, outer-sum decomposition, norms, determinants, rank, and inverses, connecting Part I's key linear-algebra identities.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Square each component

    Turn both negative and positive components into nonnegative contributions.

  2. 2

    Sum the squares

    ∑ixi2\sum_i x_i^2
  3. 3

    Take the square root for a norm

    ∥x∥2=∑ixi2\lVert x\rVert_2=\sqrt{\sum_i x_i^2}
  4. 4

    Distinguish the squared norm

    ∥x∥22=∑ixi2\lVert x\rVert_2^2=\sum_i x_i^2

Worked example

Work through the numbers — The 3–4–5 vector

A familiar Pythagorean example.

  1. 1

    Vector

    x=[3,4]Tx=[3,4]^T
  2. 2

    Sum of squares

    32+42=9+16=253^2+4^2=9+16=25
  3. 3

    Square root

    25=5\sqrt{25}=5

Result: Thus ||x||₂=5 and ||x||₂²=25.

Sanity check

Checks
  • A norm is always nonnegative.
  • Confusing a norm with its square changes units and derivative coefficients.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI3-010 Identifier: MAI3-010

Part I · Study equation

A norm summarizes many components as one magnitude
∥x∥22=xTx.\|\mathbf{x}\|_2^2=\mathbf{x}^T\mathbf{x}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How far is a vector or error from the origin?

Key idea

The L2 norm sums squared components and takes the square root. Squared norm omits the final square root.

Formula-specific lesson

Connect dot products, outer products, matrix products, norms, rank, and inverses in one numerical example

Use the same x, y, A, and B to calculate dot and outer products, Ax, AB, outer-sum decomposition, norms, determinants, rank, and inverses, connecting Part I's key linear-algebra identities.

Synchronized derivation
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Square each component

    Turn both negative and positive components into nonnegative contributions.

  2. 2

    Sum the squares

    ∑ixi2\sum_i x_i^2
  3. 3

    Take the square root for a norm

    ∥x∥2=∑ixi2\lVert x\rVert_2=\sqrt{\sum_i x_i^2}
  4. 4

    Distinguish the squared norm

    ∥x∥22=∑ixi2\lVert x\rVert_2^2=\sum_i x_i^2

Worked example

Work through the numbers — The 3–4–5 vector

A familiar Pythagorean example.

  1. 1

    Vector

    x=[3,4]Tx=[3,4]^T
  2. 2

    Sum of squares

    32+42=9+16=253^2+4^2=9+16=25
  3. 3

    Square root

    25=5\sqrt{25}=5

Result: Thus ||x||₂=5 and ||x||₂²=25.

Sanity check

Checks
  • A norm is always nonnegative.
  • Confusing a norm with its square changes units and derivative coefficients.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI3-011 Identifier: MAI3-011

A norm generalizes length and satisfies four properties.

Part I · Study equation

Identify the equation's role and symbols
f(x)≥0.f(\mathbf{x})\ge 0.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-012 Identifier: MAI3-012

Part I · Study equation

Identify the equation's role and symbols
f(x)=0  ⟺  x=0.f(\mathbf{x})=0\iff \mathbf{x}=0.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-013 Identifier: MAI3-013

Part I · Study equation

Identify the equation's role and symbols
f(tx)=∣t∣f(x).f(t\mathbf{x})=|t|f(\mathbf{x}).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-014 Identifier: MAI3-014

Part I · Study equation

Identify the equation's role and symbols
f(x+y)≤f(x)+f(y).f(\mathbf{x}+\mathbf{y})\le f(\mathbf{x})+f(\mathbf{y}).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-015 Identifier: MAI3-015

Other representative norms are included too.

Part I · Study equation

A norm summarizes many components as one magnitude
∥x∥1=∑i=1N∣xi∣.\|\mathbf{x}\|_1=\sum_{i=1}^{N}|x_i|.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How far is a vector or error from the origin?

Key idea

The L2 norm sums squared components and takes the square root. Squared norm omits the final square root.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Square each component

    Turn both negative and positive components into nonnegative contributions.

  2. 2

    Sum the squares

    ∑ixi2\sum_i x_i^2
  3. 3

    Take the square root for a norm

    ∥x∥2=∑ixi2\lVert x\rVert_2=\sqrt{\sum_i x_i^2}
  4. 4

    Distinguish the squared norm

    ∥x∥22=∑ixi2\lVert x\rVert_2^2=\sum_i x_i^2

Worked example

Work through the numbers — The 3–4–5 vector

A familiar Pythagorean example.

  1. 1

    Vector

    x=[3,4]Tx=[3,4]^T
  2. 2

    Sum of squares

    32+42=9+16=253^2+4^2=9+16=25
  3. 3

    Square root

    25=5\sqrt{25}=5

Result: Thus ||x||₂=5 and ||x||₂²=25.

Sanity check

Checks
  • A norm is always nonnegative.
  • Confusing a norm with its square changes units and derivative coefficients.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI3-016 Identifier: MAI3-016

Part I · Study equation

MSE has three steps: residuals, squares, and averaging
∥x∥∞=max⁡i∣xi∣.\|\mathbf{x}\|_\infty=\max_i |x_i|.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How far are predictions from targets on average?

Key idea

Subtract targets from predictions, square each residual, and divide by sample count. Larger errors receive greater penalties.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate predictions

    For linear regression, predictions are Xw or wᵀx.

    y^=Xw\hat y=Xw
  2. 2

    Calculate residuals

    r=y^−yr=\hat y-y
  3. 3

    Square the residuals

    Remove signs and emphasize larger errors.

    ri2r_i^2
  4. 4

    Sum and divide by sample count

    MSE=1L∑i=1Lri2\mathrm{MSE}=\frac{1}{L}\sum_{i=1}^{L}r_i^2

Worked example

Work through the numbers — MSE for two samples

Compare predictions and targets directly.

  1. 1

    Targets and predictions

    y=[3,1],y^=[2,4]y=[3,1],\quad\hat y=[2,4]
  2. 2

    Residual

    y^−y=[−1,3]\hat y-y=[-1,3]
  3. 3

    Square

    [−1,3]2=[1,9][-1,3]^2=[1,9]
  4. 4

    Average

    (1+9)/2=5(1+9)/2=5

Result: MSE is 5. The second sample's error of 3 substantially increases it.

Sanity check

Checks
  • Check whether the expression divides by sample count or is merely a sum of squares.
  • MSE has squared output units.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-017 Identifier: MAI3-017

Part I · Study equation

A norm summarizes many components as one magnitude
∥x∥p=(∑i=1N∣xi∣p)1/p,p≥1.\|\mathbf{x}\|_p= \left(\sum_{i=1}^{N}|x_i|^p\right)^{1/p},\qquad p\ge1.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How far is a vector or error from the origin?

Key idea

The L2 norm sums squared components and takes the square root. Squared norm omits the final square root.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Square each component

    Turn both negative and positive components into nonnegative contributions.

  2. 2

    Sum the squares

    ∑ixi2\sum_i x_i^2
  3. 3

    Take the square root for a norm

    ∥x∥2=∑ixi2\lVert x\rVert_2=\sqrt{\sum_i x_i^2}
  4. 4

    Distinguish the squared norm

    ∥x∥22=∑ixi2\lVert x\rVert_2^2=\sum_i x_i^2

Worked example

Work through the numbers — The 3–4–5 vector

A familiar Pythagorean example.

  1. 1

    Vector

    x=[3,4]Tx=[3,4]^T
  2. 2

    Sum of squares

    32+42=9+16=253^2+4^2=9+16=25
  3. 3

    Square root

    25=5\sqrt{25}=5

Result: Thus ||x||₂=5 and ||x||₂²=25.

Sanity check

Checks
  • A norm is always nonnegative.
  • Confusing a norm with its square changes units and derivative coefficients.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI3-018 Identifier: MAI3-018

For matrices, the Frobenius norm frequently appears.

Part I · Study equation

A norm summarizes many components as one magnitude
∥A∥F=∑i=1M∑j=1NAi,j2=tr⁡(ATA).\|A\|_F= \sqrt{\sum_{i=1}^{M}\sum_{j=1}^{N}A_{i,j}^2} = \sqrt{\operatorname{tr}(A^TA)}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How far is a vector or error from the origin?

Key idea

The L2 norm sums squared components and takes the square root. Squared norm omits the final square root.

Formula-specific lesson

Connect dot products, outer products, matrix products, norms, rank, and inverses in one numerical example

Use the same x, y, A, and B to calculate dot and outer products, Ax, AB, outer-sum decomposition, norms, determinants, rank, and inverses, connecting Part I's key linear-algebra identities.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Square each component

    Turn both negative and positive components into nonnegative contributions.

  2. 2

    Sum the squares

    ∑ixi2\sum_i x_i^2
  3. 3

    Take the square root for a norm

    ∥x∥2=∑ixi2\lVert x\rVert_2=\sqrt{\sum_i x_i^2}
  4. 4

    Distinguish the squared norm

    ∥x∥22=∑ixi2\lVert x\rVert_2^2=\sum_i x_i^2

Worked example

Work through the numbers — The 3–4–5 vector

A familiar Pythagorean example.

  1. 1

    Vector

    x=[3,4]Tx=[3,4]^T
  2. 2

    Sum of squares

    32+42=9+16=253^2+4^2=9+16=25
  3. 3

    Square root

    25=5\sqrt{25}=5

Result: Thus ||x||₂=5 and ||x||₂²=25.

Sanity check

Checks
  • A norm is always nonnegative.
  • Confusing a norm with its square changes units and derivative coefficients.

Symbols

Symbols used in this equation
SymbolMeaning
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI3-019 Identifier: MAI3-019

#4. Linear independence and rank

If one vector can be expressed as a linear combination of the others, the set is linearly dependent.

Part I · Study equation

Identify the equation's role and symbols
xN=∑i=1N−1αixi.\mathbf{x}_N=\sum_{i=1}^{N-1}\alpha_i\mathbf{x}_i.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
α\alphaLearning rate, mixture coefficient, or dual coefficient
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-020 Identifier: MAI3-020

The lecture gives these three example vectors.

Part I · Study equation

Matrix multiplication repeats row–column inner products
x1=[123],x2=[415],x3=[2−3−1].\mathbf{x}_1= \begin{bmatrix}1\\2\\3\end{bmatrix},\quad \mathbf{x}_2= \begin{bmatrix}4\\1\\5\end{bmatrix},\quad \mathbf{x}_3= \begin{bmatrix}2\\-3\\-1\end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI3-021 Identifier: MAI3-021

Part I · Study equation

Identify the equation's role and symbols
x3=−2x1+x2.\mathbf{x}_3=-2\mathbf{x}_1+\mathbf{x}_2.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-022 Identifier: MAI3-022

Basic rank properties follow.

Part I · Study equation

Identify the equation's role and symbols
rank⁡(A)≤min⁡(M,N),A∈RM×N.\operatorname{rank}(A)\le\min(M,N),\qquad A\in\mathbb{R}^{M\times N}.
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

Connect dot products, outer products, matrix products, norms, rank, and inverses in one numerical example

Use the same x, y, A, and B to calculate dot and outer products, Ax, AB, outer-sum decomposition, norms, determinants, rank, and inverses, connecting Part I's key linear-algebra identities.

Structure explanation
Loading the lesson matched to this equation and derivation step.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-023 Identifier: MAI3-023

Part I · Study equation

Identify the equation's role and symbols
rank⁡(A)=rank⁡(AT).\operatorname{rank}(A)=\operatorname{rank}(A^T).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-024 Identifier: MAI3-024

Part I · Study equation

Identify the equation's role and symbols
rank⁡(AB)≤min⁡(rank⁡(A),rank⁡(B)).\operatorname{rank}(AB)\le\min(\operatorname{rank}(A),\operatorname{rank}(B)).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-025 Identifier: MAI3-025

Part I · Study equation

Identify the equation's role and symbols
rank⁡(A+B)≤rank⁡(A)+rank⁡(B).\operatorname{rank}(A+B)\le\operatorname{rank}(A)+\operatorname{rank}(B).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-026 Identifier: MAI3-026

#5. Matrix inverse

Part I · Study equation

An inverse undoes a linear transformation
A−1A=I=AA−1.A^{-1}A=I=AA^{-1}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How can the vector before applying A be recovered?

Key idea

Using A^{-1}A=I, left-multiplying Ax=b by the inverse gives x=A^{-1}b.

Formula-specific lesson

Connect dot products, outer products, matrix products, norms, rank, and inverses in one numerical example

Use the same x, y, A, and B to calculate dot and outer products, Ax, AB, outer-sum decomposition, norms, determinants, rank, and inverses, connecting Part I's key linear-algebra identities.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check invertibility

    A classical inverse requires a square matrix with nonzero determinant.

    det⁡(A)≠0\det(A)\neq0
  2. 2

    Left-multiply both sides by the same matrix

    A−1Ax=A−1bA^{-1}Ax=A^{-1}b
  3. 3

    Replace A^{-1}A with the identity

    Ix=A−1bIx=A^{-1}b
  4. 4

    Use a linear-system solver in practice

    Numerically, solve, LU, or QR is preferable to explicitly forming an inverse.

Worked example

Work through the numbers — Checking a 2×2 inverse

Verify reversal directly with a small matrix.

  1. 1

    Matrix

    A=[2004]A=\begin{bmatrix}2&0\\0&4\end{bmatrix}
  2. 2

    Inverse

    A−1=[1/2001/4]A^{-1}=\begin{bmatrix}1/2&0\\0&1/4\end{bmatrix}
  3. 3

    Product

    A−1A=[1001]=IA^{-1}A=\begin{bmatrix}1&0\\0&1\end{bmatrix}=I

Result: Taking reciprocals of diagonal entries reverses the original scaling.

Sanity check

Checks
  • If det(A)=0, a pseudoinverse or least-squares solution is needed instead.
  • Check both AB=I and BA=I.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI3-027 Identifier: MAI3-027

Even a square matrix has no inverse if singular. For nonsingular matrices, the following properties hold.

Part I · Study equation

Identify the equation's role and symbols
(A−1)−1=A.(A^{-1})^{-1}=A.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-028 Identifier: MAI3-028

Part I · Study equation

Identify the equation's role and symbols
(AB)−1=B−1A−1.(AB)^{-1}=B^{-1}A^{-1}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-029 Identifier: MAI3-029

Part I · Study equation

Identify the equation's role and symbols
(A−1)T=(AT)−1≡A−T.(A^{-1})^T=(A^T)^{-1}\equiv A^{-T}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-030 Identifier: MAI3-030

#6. Orthogonality and orthogonal matrices

Part I · Study equation

Identify the equation's role and symbols
xTy=0.\mathbf{x}^T\mathbf{y}=0.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-031 Identifier: MAI3-031

Part I · Study equation

A norm summarizes many components as one magnitude
∥x∥2=1.\|\mathbf{x}\|_2=1.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How far is a vector or error from the origin?

Key idea

The L2 norm sums squared components and takes the square root. Squared norm omits the final square root.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Square each component

    Turn both negative and positive components into nonnegative contributions.

  2. 2

    Sum the squares

    ∑ixi2\sum_i x_i^2
  3. 3

    Take the square root for a norm

    ∥x∥2=∑ixi2\lVert x\rVert_2=\sqrt{\sum_i x_i^2}
  4. 4

    Distinguish the squared norm

    ∥x∥22=∑ixi2\lVert x\rVert_2^2=\sum_i x_i^2

Worked example

Work through the numbers — The 3–4–5 vector

A familiar Pythagorean example.

  1. 1

    Vector

    x=[3,4]Tx=[3,4]^T
  2. 2

    Sum of squares

    32+42=9+16=253^2+4^2=9+16=25
  3. 3

    Square root

    25=5\sqrt{25}=5

Result: Thus ||x||₂=5 and ||x||₂²=25.

Sanity check

Checks
  • A norm is always nonnegative.
  • Confusing a norm with its square changes units and derivative coefficients.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI3-032 Identifier: MAI3-032

A square orthogonal matrix U has orthonormal columns and satisfies:

Part I · Study equation

Identify the equation's role and symbols
UTU=I=UUT.U^TU=I=UU^T.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-033 Identifier: MAI3-033

If only a rectangular matrix’s columns are orthonormal, just one multiplication order yields an identity.

Part I · Study equation

Identify the equation's role and symbols
UTU=I,UUT≠I(U∈RM×N, N<M).U^TU=I,\qquad UU^T\neq I\quad (U\in\mathbb{R}^{M\times N},\,N<M).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-034 Identifier: MAI3-034

Orthogonal transformations preserve Euclidean length.

Part I · Study equation

A norm summarizes many components as one magnitude
∥Ux∥2=∥x∥2.\|U\mathbf{x}\|_2=\|\mathbf{x}\|_2.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How far is a vector or error from the origin?

Key idea

The L2 norm sums squared components and takes the square root. Squared norm omits the final square root.

Formula-specific lesson

Connect dot products, outer products, matrix products, norms, rank, and inverses in one numerical example

Use the same x, y, A, and B to calculate dot and outer products, Ax, AB, outer-sum decomposition, norms, determinants, rank, and inverses, connecting Part I's key linear-algebra identities.

Structure explanation
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Square each component

    Turn both negative and positive components into nonnegative contributions.

  2. 2

    Sum the squares

    ∑ixi2\sum_i x_i^2
  3. 3

    Take the square root for a norm

    ∥x∥2=∑ixi2\lVert x\rVert_2=\sqrt{\sum_i x_i^2}
  4. 4

    Distinguish the squared norm

    ∥x∥22=∑ixi2\lVert x\rVert_2^2=\sum_i x_i^2

Worked example

Work through the numbers — The 3–4–5 vector

A familiar Pythagorean example.

  1. 1

    Vector

    x=[3,4]Tx=[3,4]^T
  2. 2

    Sum of squares

    32+42=9+16=253^2+4^2=9+16=25
  3. 3

    Square root

    25=5\sqrt{25}=5

Result: Thus ||x||₂=5 and ||x||₂²=25.

Sanity check

Checks
  • A norm is always nonnegative.
  • Confusing a norm with its square changes units and derivative coefficients.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI3-035 Identifier: MAI3-035

#7. Determinant

The determinant’s absolute value represents the volume of the parallelotope formed by the row vectors.

Part I · Study equation

Identify the equation's role and symbols
S={v∈RN:v=∑i=1Nαiai,0≤αi≤1}.S= \left\{ \mathbf{v}\in\mathbb{R}^N: \mathbf{v}=\sum_{i=1}^{N}\alpha_i\mathbf{a}_i,\quad 0\le\alpha_i\le1 \right\}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
α\alphaLearning rate, mixture coefficient, or dual coefficient
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-036 Identifier: MAI3-036

The lecture’s 2×2 example follows.

Part I · Study equation

Matrix multiplication repeats row–column inner products
A=[1332].A= \begin{bmatrix} 1&3\\ 3&2 \end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI3-037 Identifier: MAI3-037

Part I · Study equation

Matrix multiplication repeats row–column inner products
a1=[13],a2=[32].\mathbf{a}_1= \begin{bmatrix}1\\3\end{bmatrix},\qquad \mathbf{a}_2= \begin{bmatrix}3\\2\end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI3-038 Identifier: MAI3-038

Part I · Study equation

Identify the equation's role and symbols
det⁡(A)=∣A∣=−7,∣det⁡(A)∣=7.\det(A)=|A|=-7,\qquad |\det(A)|=7.
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

Connect dot products, outer products, matrix products, norms, rank, and inverses in one numerical example

Use the same x, y, A, and B to calculate dot and outer products, Ax, AB, outer-sum decomposition, norms, determinants, rank, and inverses, connecting Part I's key linear-algebra identities.

Interactive
Loading the lesson matched to this equation and derivation step.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-039 Identifier: MAI3-039

The key determinant properties are listed in full.

Part I · Study equation

Identify the equation's role and symbols
∣I∣=1.|I|=1.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-040 Identifier: MAI3-040

Part I · Study equation

Matrix multiplication repeats row–column inner products
det⁡[ta1Ta2T⋮aNT]=t∣A∣.\det \begin{bmatrix} t\mathbf{a}_1^T\\ \mathbf{a}_2^T\\ \vdots\\ \mathbf{a}_N^T \end{bmatrix} =t|A|.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI3-041 Identifier: MAI3-041

Part I · Study equation

Matrix multiplication repeats row–column inner products
det⁡[a2Ta1T⋮aNT]=−∣A∣.\det \begin{bmatrix} \mathbf{a}_2^T\\ \mathbf{a}_1^T\\ \vdots\\ \mathbf{a}_N^T \end{bmatrix} =-|A|.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI3-042 Identifier: MAI3-042

Part I · Study equation

Identify the equation's role and symbols
∣A∣=∣AT∣.|A|=|A^T|.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-043 Identifier: MAI3-043

Part I · Study equation

Identify the equation's role and symbols
∣AB∣=∣A∣∣B∣.|AB|=|A||B|.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-044 Identifier: MAI3-044

Part I · Study equation

Identify the equation's role and symbols
∣A∣=0  ⟺  A is singular.|A|=0\iff A\ \text{is singular}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-045 Identifier: MAI3-045

Part I · Study equation

Identify the equation's role and symbols
∣A−1∣=1∣A∣.|A^{-1}|=\frac{1}{|A|}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI3-046 Identifier: MAI3-046

#8. Numerical computation in 2026

An important practical refinement is to distinguish mathematical definitions from numerical code.

  • Inverses matter mathematically, but explicitly forming inv(A) @ b every time is not a good default computational habit. For linear systems, solve is more direct.
  • Small determinants can underflow and large ones overflow, making the log determinant slogdet useful.
  • Treat rank as numerical rank depending on floating-point tolerance, rather than exact symbolic rank.

Current PyTorch’s torch.linalg includes solve, slogdet, matrix_rank, qr, svd, and eigh.

#Part 1.3: Linear Algebra III — Quadratic Forms, Eigenvalues, and Derivatives

#1. Quadratic forms

Part I · Study equation

Solving simultaneous equations means satisfying all constraints together
xTAx=∑i=1Nxi(Ax)i=∑i=1N∑j=1NAi,jxixj.\mathbf{x}^TA\mathbf{x} = \sum_{i=1}^{N}x_i(A\mathbf{x})_i = \sum_{i=1}^{N}\sum_{j=1}^{N}A_{i,j}x_i x_j.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

Which unknown vector makes every row's equation hold simultaneously?

Key idea

Read each row as an equation and solve using elimination, an inverse, or a factorization.

Formula-specific lesson

Quadratic forms, eigenvalues, covariance, and PCA axes

Use one symmetric matrix and a 2D sample cloud to calculate quadratic forms, eigenpairs, trace/determinant identities, covariance symmetry, and principal axes together.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Expand into rowwise equations

    One matrix row dotted with the unknown vector produces one right-hand-side value.

    Ai,:x=biA_{i,:}x=b_i
  2. 2

    Eliminate a variable

    Add or subtract equations to reduce the unknown count. LU or QR factorizations play this role in larger problems.

  3. 3

    Back-substitute the remaining value

    Insert the solved value into an original equation to obtain other unknowns.

  4. 4

    Substitute into every equation again

    Check that the calculated solution satisfies every row.

    Ax=?bAx\stackrel{?}=b

Worked example

Work through the numbers — Solving a 2×2 system directly

Think of finding the intersection of two lines.

  1. 1

    Problem

    2x+y=5,x−y=12x+y=5,\qquad x-y=1
  2. 2

    Add equations to eliminate y

    3x=6⇒x=23x=6\Rightarrow x=2
  3. 3

    Back-substitute into the second equation

    2−y=1⇒y=12-y=1\Rightarrow y=1
  4. 4

    Check

    2(2)+1=5,2−1=12(2)+1=5,\qquad 2-1=1

Result: The solution is therefore (x,y)=(2,1).

Sanity check

Checks
  • Ax on the left and b on the right must have the same dimension.
  • The inverse expression applies directly only when A is square and invertible.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI4-001 Identifier: MAI4-001

Only A’s symmetric part contributes to a quadratic form.

Part I · Study equation

Solving simultaneous equations means satisfying all constraints together
xTAx=(xTAx)T=xTATx=xT(12A+12AT)x.\mathbf{x}^TA\mathbf{x} = (\mathbf{x}^TA\mathbf{x})^T = \mathbf{x}^TA^T\mathbf{x} = \mathbf{x}^T\left(\frac12A+\frac12A^T\right)\mathbf{x}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which unknown vector makes every row's equation hold simultaneously?

Key idea

Read each row as an equation and solve using elimination, an inverse, or a factorization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Expand into rowwise equations

    One matrix row dotted with the unknown vector produces one right-hand-side value.

    Ai,:x=biA_{i,:}x=b_i
  2. 2

    Eliminate a variable

    Add or subtract equations to reduce the unknown count. LU or QR factorizations play this role in larger problems.

  3. 3

    Back-substitute the remaining value

    Insert the solved value into an original equation to obtain other unknowns.

  4. 4

    Substitute into every equation again

    Check that the calculated solution satisfies every row.

    Ax=?bAx\stackrel{?}=b

Worked example

Work through the numbers — Solving a 2×2 system directly

Think of finding the intersection of two lines.

  1. 1

    Problem

    2x+y=5,x−y=12x+y=5,\qquad x-y=1
  2. 2

    Add equations to eliminate y

    3x=6⇒x=23x=6\Rightarrow x=2
  3. 3

    Back-substitute into the second equation

    2−y=1⇒y=12-y=1\Rightarrow y=1
  4. 4

    Check

    2(2)+1=5,2−1=12(2)+1=5,\qquad 2-1=1

Result: The solution is therefore (x,y)=(2,1).

Sanity check

Checks
  • Ax on the left and b on the right must have the same dimension.
  • The inverse expression applies directly only when A is square and invertible.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI4-002 Identifier: MAI4-002

#2. Positive definiteness and semidefiniteness

Part I · Study equation

Solving simultaneous equations means satisfying all constraints together
A≻0  ⟺  xTAx>0for all x≠0.A\succ0 \iff \mathbf{x}^TA\mathbf{x}>0 \quad\text{for all }\mathbf{x}\neq0.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which unknown vector makes every row's equation hold simultaneously?

Key idea

Read each row as an equation and solve using elimination, an inverse, or a factorization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Expand into rowwise equations

    One matrix row dotted with the unknown vector produces one right-hand-side value.

    Ai,:x=biA_{i,:}x=b_i
  2. 2

    Eliminate a variable

    Add or subtract equations to reduce the unknown count. LU or QR factorizations play this role in larger problems.

  3. 3

    Back-substitute the remaining value

    Insert the solved value into an original equation to obtain other unknowns.

  4. 4

    Substitute into every equation again

    Check that the calculated solution satisfies every row.

    Ax=?bAx\stackrel{?}=b

Worked example

Work through the numbers — Solving a 2×2 system directly

Think of finding the intersection of two lines.

  1. 1

    Problem

    2x+y=5,x−y=12x+y=5,\qquad x-y=1
  2. 2

    Add equations to eliminate y

    3x=6⇒x=23x=6\Rightarrow x=2
  3. 3

    Back-substitute into the second equation

    2−y=1⇒y=12-y=1\Rightarrow y=1
  4. 4

    Check

    2(2)+1=5,2−1=12(2)+1=5,\qquad 2-1=1

Result: The solution is therefore (x,y)=(2,1).

Sanity check

Checks
  • Ax on the left and b on the right must have the same dimension.
  • The inverse expression applies directly only when A is square and invertible.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI4-003 Identifier: MAI4-003

Part I · Study equation

Solving simultaneous equations means satisfying all constraints together
A⪰0  ⟺  xTAx≥0for all x.A\succeq0 \iff \mathbf{x}^TA\mathbf{x}\ge0 \quad\text{for all }\mathbf{x}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which unknown vector makes every row's equation hold simultaneously?

Key idea

Read each row as an equation and solve using elimination, an inverse, or a factorization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Expand into rowwise equations

    One matrix row dotted with the unknown vector produces one right-hand-side value.

    Ai,:x=biA_{i,:}x=b_i
  2. 2

    Eliminate a variable

    Add or subtract equations to reduce the unknown count. LU or QR factorizations play this role in larger problems.

  3. 3

    Back-substitute the remaining value

    Insert the solved value into an original equation to obtain other unknowns.

  4. 4

    Substitute into every equation again

    Check that the calculated solution satisfies every row.

    Ax=?bAx\stackrel{?}=b

Worked example

Work through the numbers — Solving a 2×2 system directly

Think of finding the intersection of two lines.

  1. 1

    Problem

    2x+y=5,x−y=12x+y=5,\qquad x-y=1
  2. 2

    Add equations to eliminate y

    3x=6⇒x=23x=6\Rightarrow x=2
  3. 3

    Back-substitute into the second equation

    2−y=1⇒y=12-y=1\Rightarrow y=1
  4. 4

    Check

    2(2)+1=5,2−1=12(2)+1=5,\qquad 2-1=1

Result: The solution is therefore (x,y)=(2,1).

Sanity check

Checks
  • Ax on the left and b on the right must have the same dimension.
  • The inverse expression applies directly only when A is square and invertible.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI4-004 Identifier: MAI4-004

Part I · Study equation

Solving simultaneous equations means satisfying all constraints together
A≺0  ⟺  xTAx<0for all x≠0.A\prec0 \iff \mathbf{x}^TA\mathbf{x}<0 \quad\text{for all }\mathbf{x}\neq0.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which unknown vector makes every row's equation hold simultaneously?

Key idea

Read each row as an equation and solve using elimination, an inverse, or a factorization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Expand into rowwise equations

    One matrix row dotted with the unknown vector produces one right-hand-side value.

    Ai,:x=biA_{i,:}x=b_i
  2. 2

    Eliminate a variable

    Add or subtract equations to reduce the unknown count. LU or QR factorizations play this role in larger problems.

  3. 3

    Back-substitute the remaining value

    Insert the solved value into an original equation to obtain other unknowns.

  4. 4

    Substitute into every equation again

    Check that the calculated solution satisfies every row.

    Ax=?bAx\stackrel{?}=b

Worked example

Work through the numbers — Solving a 2×2 system directly

Think of finding the intersection of two lines.

  1. 1

    Problem

    2x+y=5,x−y=12x+y=5,\qquad x-y=1
  2. 2

    Add equations to eliminate y

    3x=6⇒x=23x=6\Rightarrow x=2
  3. 3

    Back-substitute into the second equation

    2−y=1⇒y=12-y=1\Rightarrow y=1
  4. 4

    Check

    2(2)+1=5,2−1=12(2)+1=5,\qquad 2-1=1

Result: The solution is therefore (x,y)=(2,1).

Sanity check

Checks
  • Ax on the left and b on the right must have the same dimension.
  • The inverse expression applies directly only when A is square and invertible.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI4-005 Identifier: MAI4-005

Part I · Study equation

Solving simultaneous equations means satisfying all constraints together
A⪯0  ⟺  xTAx≤0for all x.A\preceq0 \iff \mathbf{x}^TA\mathbf{x}\le0 \quad\text{for all }\mathbf{x}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which unknown vector makes every row's equation hold simultaneously?

Key idea

Read each row as an equation and solve using elimination, an inverse, or a factorization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Expand into rowwise equations

    One matrix row dotted with the unknown vector produces one right-hand-side value.

    Ai,:x=biA_{i,:}x=b_i
  2. 2

    Eliminate a variable

    Add or subtract equations to reduce the unknown count. LU or QR factorizations play this role in larger problems.

  3. 3

    Back-substitute the remaining value

    Insert the solved value into an original equation to obtain other unknowns.

  4. 4

    Substitute into every equation again

    Check that the calculated solution satisfies every row.

    Ax=?bAx\stackrel{?}=b

Worked example

Work through the numbers — Solving a 2×2 system directly

Think of finding the intersection of two lines.

  1. 1

    Problem

    2x+y=5,x−y=12x+y=5,\qquad x-y=1
  2. 2

    Add equations to eliminate y

    3x=6⇒x=23x=6\Rightarrow x=2
  3. 3

    Back-substitute into the second equation

    2−y=1⇒y=12-y=1\Rightarrow y=1
  4. 4

    Check

    2(2)+1=5,2−1=12(2)+1=5,\qquad 2-1=1

Result: The solution is therefore (x,y)=(2,1).

Sanity check

Checks
  • Ax on the left and b on the right must have the same dimension.
  • The inverse expression applies directly only when A is square and invertible.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI4-006 Identifier: MAI4-006

Indefinite means positive in some directions and negative in others.

Part I · Study equation

Solving simultaneous equations means satisfying all constraints together
∃x1,x2:x1TAx1>0,x2TAx2<0.\exists\mathbf{x}_1,\mathbf{x}_2: \quad \mathbf{x}_1^TA\mathbf{x}_1>0,\qquad \mathbf{x}_2^TA\mathbf{x}_2<0.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which unknown vector makes every row's equation hold simultaneously?

Key idea

Read each row as an equation and solve using elimination, an inverse, or a factorization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Expand into rowwise equations

    One matrix row dotted with the unknown vector produces one right-hand-side value.

    Ai,:x=biA_{i,:}x=b_i
  2. 2

    Eliminate a variable

    Add or subtract equations to reduce the unknown count. LU or QR factorizations play this role in larger problems.

  3. 3

    Back-substitute the remaining value

    Insert the solved value into an original equation to obtain other unknowns.

  4. 4

    Substitute into every equation again

    Check that the calculated solution satisfies every row.

    Ax=?bAx\stackrel{?}=b

Worked example

Work through the numbers — Solving a 2×2 system directly

Think of finding the intersection of two lines.

  1. 1

    Problem

    2x+y=5,x−y=12x+y=5,\qquad x-y=1
  2. 2

    Add equations to eliminate y

    3x=6⇒x=23x=6\Rightarrow x=2
  3. 3

    Back-substitute into the second equation

    2−y=1⇒y=12-y=1\Rightarrow y=1
  4. 4

    Check

    2(2)+1=5,2−1=12(2)+1=5,\qquad 2-1=1

Result: The solution is therefore (x,y)=(2,1).

Sanity check

Checks
  • Ax on the left and b on the right must have the same dimension.
  • The inverse expression applies directly only when A is square and invertible.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI4-007 Identifier: MAI4-007

#3. Eigenvalues and eigenvectors

Part I · Study equation

Solving simultaneous equations means satisfying all constraints together
Ax=λx,x≠0.A\mathbf{x}=\lambda\mathbf{x},\qquad \mathbf{x}\neq0.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

Which unknown vector makes every row's equation hold simultaneously?

Key idea

Read each row as an equation and solve using elimination, an inverse, or a factorization.

Formula-specific lesson

Quadratic forms, eigenvalues, covariance, and PCA axes

Use one symmetric matrix and a 2D sample cloud to calculate quadratic forms, eigenpairs, trace/determinant identities, covariance symmetry, and principal axes together.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Expand into rowwise equations

    One matrix row dotted with the unknown vector produces one right-hand-side value.

    Ai,:x=biA_{i,:}x=b_i
  2. 2

    Eliminate a variable

    Add or subtract equations to reduce the unknown count. LU or QR factorizations play this role in larger problems.

  3. 3

    Back-substitute the remaining value

    Insert the solved value into an original equation to obtain other unknowns.

  4. 4

    Substitute into every equation again

    Check that the calculated solution satisfies every row.

    Ax=?bAx\stackrel{?}=b

Worked example

Work through the numbers — Solving a 2×2 system directly

Think of finding the intersection of two lines.

  1. 1

    Problem

    2x+y=5,x−y=12x+y=5,\qquad x-y=1
  2. 2

    Add equations to eliminate y

    3x=6⇒x=23x=6\Rightarrow x=2
  3. 3

    Back-substitute into the second equation

    2−y=1⇒y=12-y=1\Rightarrow y=1
  4. 4

    Check

    2(2)+1=5,2−1=12(2)+1=5,\qquad 2-1=1

Result: The solution is therefore (x,y)=(2,1).

Sanity check

Checks
  • Ax on the left and b on the right must have the same dimension.
  • The inverse expression applies directly only when A is square and invertible.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
λ\lambdaRegularization strength or eigenvalue

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI4-008 Identifier: MAI4-008

Eigenvalues connect directly with trace, determinant, and rank.

Part I · Study equation

Identify the equation's role and symbols
tr⁡(A)=∑i=1Nλi.\operatorname{tr}(A)=\sum_{i=1}^{N}\lambda_i.
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

Quadratic forms, eigenvalues, covariance, and PCA axes

Use one symmetric matrix and a 2D sample cloud to calculate quadratic forms, eigenpairs, trace/determinant identities, covariance symmetry, and principal axes together.

Synchronized derivation
Loading the lesson matched to this equation and derivation step.

Symbols

Symbols used in this equation
SymbolMeaning
λ\lambdaRegularization strength or eigenvalue
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI4-009 Identifier: MAI4-009

Part I · Study equation

Naive Bayes multiplies feature likelihoods under conditional independence
∣A∣=∏i=1Nλi.|A|=\prod_{i=1}^{N}\lambda_i.
Explanation and calculations 5 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

Which class scores highest after observing several features?

Key idea

Multiply class priors by successive conditional feature likelihoods and compare classes.

Formula-specific lesson

Quadratic forms, eigenvalues, covariance, and PCA axes

Use one symmetric matrix and a 2D sample cloud to calculate quadratic forms, eigenpairs, trace/determinant identities, covariance symmetry, and principal axes together.

Synchronized derivation
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Prepare class priors

    P(Sc)P(S_c)
  2. 2

    Calculate conditional feature likelihoods

    p(xn∣Sc)p(x_n\mid S_c)
  3. 3

    Multiply all terms

    scorec=P(Sc)∏np(xn∣Sc)\text{score}_c=P(S_c)\prod_np(x_n\mid S_c)
  4. 4

    Use sums of logarithms for numerical stability

    log⁡scorec=log⁡P(Sc)+∑nlog⁡p(xn∣Sc)\log\text{score}_c=\log P(S_c)+\sum_n\log p(x_n\mid S_c)
  5. 5

    Select the highest-scoring class

    c∗=arg⁡max⁡cscorecc^*=\arg\max_c\text{score}_c

Worked example

Work through the numbers — Two features and two classes

S₁: prior=.6, likelihoods=.8,.5; S₂: prior=.4, likelihoods=.3,.9.

  1. 1

    S₁ score

    0.6⋅0.8⋅0.5=0.240.6\cdot0.8\cdot0.5=0.24
  2. 2

    S₂ score

    0.4⋅0.3⋅0.9=0.1080.4\cdot0.3\cdot0.9=0.108

Result: Since 0.24>0.108, select S₁.

Sanity check

Checks
  • A general MAP expression includes class prior P(S_c).
  • Many multiplied features can underflow, so use log-domain calculations.

Symbols

Symbols used in this equation
SymbolMeaning
λ\lambdaRegularization strength or eigenvalue
∏\prodMultiply all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI4-010 Identifier: MAI4-010

Part I · Study equation

Identify the equation's role and symbols
rank⁡(A)=#{i:λi≠0}.\operatorname{rank}(A)=\#\{i:\lambda_i\neq0\}.
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

Quadratic forms, eigenvalues, covariance, and PCA axes

Use one symmetric matrix and a 2D sample cloud to calculate quadratic forms, eigenpairs, trace/determinant identities, covariance symmetry, and principal axes together.

Structure explanation
Loading the lesson matched to this equation and derivation step.

Symbols

Symbols used in this equation
SymbolMeaning
λ\lambdaRegularization strength or eigenvalue

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI4-011 Identifier: MAI4-011

Part I · Study equation

Identify the equation's role and symbols
A−1xi=1λixi.A^{-1}\mathbf{x}_i=\frac{1}{\lambda_i}\mathbf{x}_i.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
λ\lambdaRegularization strength or eigenvalue

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI4-012 Identifier: MAI4-012

Part I · Study equation

Identify the equation's role and symbols
D=diag⁡(d1,…,dN)⟹λi=di.D=\operatorname{diag}(d_1,\ldots,d_N) \quad\Longrightarrow\quad \lambda_i=d_i.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
λ\lambdaRegularization strength or eigenvalue
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI4-013 Identifier: MAI4-013

Collecting all eigenvector equations gives:

Part I · Study equation

Identify the equation's role and symbols
AX=XΛ.AX=X\Lambda.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI4-014 Identifier: MAI4-014

Part I · Study equation

Matrix multiplication repeats row–column inner products
X=[x1x2⋯xN],Λ=diag⁡(λ1,…,λN).X= \begin{bmatrix}\mathbf{x}_1&\mathbf{x}_2&\cdots&\mathbf{x}_N\end{bmatrix}, \qquad \Lambda=\operatorname{diag}(\lambda_1,\ldots,\lambda_N).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
λ\lambdaRegularization strength or eigenvalue
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI4-015 Identifier: MAI4-015

If X is invertible, A is diagonalizable.

Part I · Study equation

Identify the equation's role and symbols
A=XΛX−1.A=X\Lambda X^{-1}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
XXData matrix collecting samples
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI4-016 Identifier: MAI4-016

#4. Eigendecomposition of symmetric matrices

A symmetric matrix has real eigenvalues and can have orthonormal eigenvectors.

Part I · Study equation

Identify the equation's role and symbols
A=UΛUT.A=U\Lambda U^T.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI4-017 Identifier: MAI4-017

This structure lets eigenvalue signs determine definiteness.

Part I · Study equation

Solving simultaneous equations means satisfying all constraints together
xTAx=xTUΛUTx=yTΛy=∑i=1Nλiyi2.\mathbf{x}^TA\mathbf{x} = \mathbf{x}^TU\Lambda U^T\mathbf{x} = \mathbf{y}^T\Lambda\mathbf{y} = \sum_{i=1}^{N}\lambda_i y_i^2.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which unknown vector makes every row's equation hold simultaneously?

Key idea

Read each row as an equation and solve using elimination, an inverse, or a factorization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Expand into rowwise equations

    One matrix row dotted with the unknown vector produces one right-hand-side value.

    Ai,:x=biA_{i,:}x=b_i
  2. 2

    Eliminate a variable

    Add or subtract equations to reduce the unknown count. LU or QR factorizations play this role in larger problems.

  3. 3

    Back-substitute the remaining value

    Insert the solved value into an original equation to obtain other unknowns.

  4. 4

    Substitute into every equation again

    Check that the calculated solution satisfies every row.

    Ax=?bAx\stackrel{?}=b

Worked example

Work through the numbers — Solving a 2×2 system directly

Think of finding the intersection of two lines.

  1. 1

    Problem

    2x+y=5,x−y=12x+y=5,\qquad x-y=1
  2. 2

    Add equations to eliminate y

    3x=6⇒x=23x=6\Rightarrow x=2
  3. 3

    Back-substitute into the second equation

    2−y=1⇒y=12-y=1\Rightarrow y=1
  4. 4

    Check

    2(2)+1=5,2−1=12(2)+1=5,\qquad 2-1=1

Result: The solution is therefore (x,y)=(2,1).

Sanity check

Checks
  • Ax on the left and b on the right must have the same dimension.
  • The inverse expression applies directly only when A is square and invertible.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector
λ\lambdaRegularization strength or eigenvalue
∑\sumSum over all specified terms
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI4-018 Identifier: MAI4-018

Part I · Study equation

Identify the equation's role and symbols
y=UTx.\mathbf{y}=U^T\mathbf{x}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI4-019 Identifier: MAI4-019

#5. Gradients with respect to matrices

For a scalar-valued function of a matrix, the gradient is a matrix of partial derivatives matching the input’s shape.

Part I · Study equation

A gradient describes the direction and rate of loss change
∇Af(A)∈RM×N,(∇Af(A))i,j=∂f(A)∂Ai,j.\nabla_A f(A)\in\mathbb{R}^{M\times N},\qquad \left(\nabla_A f(A)\right)_{i,j} = \frac{\partial f(A)}{\partial A_{i,j}}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
∇\nablaA gradient collecting derivatives in each variable direction
∂\partialPartial derivative with respect to a particular variable

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI4-020 Identifier: MAI4-020

For vector inputs, this becomes the familiar gradient.

Part I · Study equation

A gradient describes the direction and rate of loss change
∇xf(x)=[∂f∂x1∂f∂x2⋮∂f∂xN].\nabla_{\mathbf{x}}f(\mathbf{x}) = \begin{bmatrix} \frac{\partial f}{\partial x_1}\\ \frac{\partial f}{\partial x_2}\\ \vdots\\ \frac{\partial f}{\partial x_N} \end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∇\nablaA gradient collecting derivatives in each variable direction
∂\partialPartial derivative with respect to a particular variable

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI4-021 Identifier: MAI4-021

Linearity is preserved as well.

Part I · Study equation

A gradient describes the direction and rate of loss change
∇x(f(x)+g(x))=∇xf(x)+∇xg(x).\nabla_{\mathbf{x}}\big(f(\mathbf{x})+g(\mathbf{x})\big) = \nabla_{\mathbf{x}}f(\mathbf{x})+\nabla_{\mathbf{x}}g(\mathbf{x}).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∇\nablaA gradient collecting derivatives in each variable direction

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI4-022 Identifier: MAI4-022

Part I · Study equation

A gradient describes the direction and rate of loss change
∇x(tf(x))=t∇xf(x).\nabla_{\mathbf{x}}\big(tf(\mathbf{x})\big) = t\nabla_{\mathbf{x}}f(\mathbf{x}).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∇\nablaA gradient collecting derivatives in each variable direction

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI4-023 Identifier: MAI4-023

#6. Hessian

Part I · Study equation

A gradient describes the direction and rate of loss change
∇x2f(x)∈RN×N,(∇x2f(x))i,j=∂2f(x)∂xi∂xj.\nabla_{\mathbf{x}}^2 f(\mathbf{x})\in\mathbb{R}^{N\times N},\qquad \left(\nabla_{\mathbf{x}}^2 f(\mathbf{x})\right)_{i,j} = \frac{\partial^2 f(\mathbf{x})}{\partial x_i\partial x_j}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∇\nablaA gradient collecting derivatives in each variable direction
∂\partialPartial derivative with respect to a particular variable

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI4-024 Identifier: MAI4-024

For sufficiently smooth functions, mixed partial derivatives agree and the Hessian is symmetric.

Part I · Study equation

A gradient describes the direction and rate of loss change
∂2f(x)∂xi∂xj=∂2f(x)∂xj∂xi.\frac{\partial^2f(\mathbf{x})}{\partial x_i\partial x_j} = \frac{\partial^2f(\mathbf{x})}{\partial x_j\partial x_i}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∂\partialPartial derivative with respect to a particular variable

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI4-025 Identifier: MAI4-025

#7. Derivative formulas for linear and quadratic functions

Part I · Study equation

A gradient describes the direction and rate of loss change
∇xbTx=b.\nabla_{\mathbf{x}}\mathbf{b}^T\mathbf{x}=\mathbf{b}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∇\nablaA gradient collecting derivatives in each variable direction

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI4-026 Identifier: MAI4-026

Part I · Study equation

A gradient describes the direction and rate of loss change
∇xxTAx=2Ax(A=AT).\nabla_{\mathbf{x}}\mathbf{x}^TA\mathbf{x}=2A\mathbf{x} \qquad(A=A^T).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∇\nablaA gradient collecting derivatives in each variable direction

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI4-027 Identifier: MAI4-027

Part I · Study equation

A gradient describes the direction and rate of loss change
∇x2xTAx=2A(A=AT).\nabla_{\mathbf{x}}^2\mathbf{x}^TA\mathbf{x}=2A \qquad(A=A^T).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∇\nablaA gradient collecting derivatives in each variable direction

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI4-028 Identifier: MAI4-028

#8. Automatic differentiation in 2026

PyTorch can calculate these gradients, Jacobians, and Hessians without manual expansion. Autodifferentiation does not eliminate mathematics: without understanding shape, symmetry, positive definiteness, and conditioning, results are easy to misinterpret.

As of the source’s August 2026 update, PyTorch 2.13’s torch.func directly provides:

  • grad
  • vjp, jvp
  • jacrev, jacfwd
  • hessian
  • vmap

Hessians involve second derivatives, so memory and compute can grow rapidly. Large-scale models often use Hessian–vector products rather than a full Hessian.

#Part 1.4: Perceptrons, Convolution, and DNNs

#1. From biological neurons to perceptrons

A perceptron does not faithfully reproduce a biological neuron. It abstracts a computational structure: inputs → weighted sum → nonlinear or threshold function → output.

One neuron in a multilayer network can be written using the lecture diagram’s equation as follows.

Part I · Study equation

Probability calculations proceed through event definitions, conditions, and sum/product/normalization rules
zi(ℓ)=∑j=1nℓ−1wi,j(ℓ)aj(ℓ−1)+bi(ℓ).z_i^{(\ell)} = \sum_{j=1}^{n_{\ell-1}} w_{i,j}^{(\ell)}a_j^{(\ell-1)} +b_i^{(\ell)}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which event or observation interval's probability is required?

Key idea

Define the event, add mutually exclusive cases, and multiply chained conditions using conditional probabilities.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Translate events and conditions into words

    P(A|B) is the probability of A given B.

  2. 2

    Choose a sum rule or product rule

    Add mutually exclusive alternatives; multiply sequential conditions.

  3. 3

    Use total probability for the denominator if needed

  4. 4

    Check 0≤P≤1 and total probability 1

Worked example

Work through the numbers — Conditional probability

P(A∩B)=0.15 and P(B)=0.30.

  1. 1

    Apply the definition

    P(A∣B)=P(A∩B)P(B)P(A\mid B)=\frac{P(A\cap B)}{P(B)}
  2. 2

    Substitute values

    0.15/0.30=0.50.15/0.30=0.5

Result: Among cases where B occurs, A occurs half the time.

Sanity check

Checks
  • The conditioning event's probability in the denominator must be nonzero.
  • Distinguish density p(x) from event probability P(X∈A).

Symbols

Symbols used in this equation
SymbolMeaning
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI5-001 Identifier: MAI5-001

Part I · Study equation

Identify the equation's role and symbols
ai(ℓ)=h ⁣(zi(ℓ)).a_i^{(\ell)}=h\!\left(z_i^{(\ell)}\right).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI5-002 Identifier: MAI5-002

#2. From a two-dimensional line to an N-dimensional hyperplane

A two-dimensional line can be written as:

Part I · Study equation

Identify the equation's role and symbols
y=ax+c.y=ax+c.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI5-003 Identifier: MAI5-003

Rearranging into a general linear boundary gives:

Part I · Study equation

Identify the equation's role and symbols
w1x1+w2x2+b=0.w_1x_1+w_2x_2+b=0.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI5-004 Identifier: MAI5-004

In N dimensions, it becomes a hyperplane.

Part I · Study equation

Probability calculations proceed through event definitions, conditions, and sum/product/normalization rules
∑n=1Nwnxn+b=0.\sum_{n=1}^{N}w_nx_n+b=0.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which event or observation interval's probability is required?

Key idea

Define the event, add mutually exclusive cases, and multiply chained conditions using conditional probabilities.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Translate events and conditions into words

    P(A|B) is the probability of A given B.

  2. 2

    Choose a sum rule or product rule

    Add mutually exclusive alternatives; multiply sequential conditions.

  3. 3

    Use total probability for the denominator if needed

  4. 4

    Check 0≤P≤1 and total probability 1

Worked example

Work through the numbers — Conditional probability

P(A∩B)=0.15 and P(B)=0.30.

  1. 1

    Apply the definition

    P(A∣B)=P(A∩B)P(B)P(A\mid B)=\frac{P(A\cap B)}{P(B)}
  2. 2

    Substitute values

    0.15/0.30=0.50.15/0.30=0.5

Result: Among cases where B occurs, A occurs half the time.

Sanity check

Checks
  • The conditioning event's probability in the denominator must be nonzero.
  • Distinguish density p(x) from event probability P(X∈A).

Symbols

Symbols used in this equation
SymbolMeaning
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI5-005 Identifier: MAI5-005

Part I · Study equation

Identify the equation's role and symbols
wTx+b=0.\mathbf{w}^T\mathbf{x}+b=0.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
w\mathbf{w}Learned weight vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI5-006 Identifier: MAI5-006

#3. The perceptron’s discriminant function

Part I · Study equation

A hyperplane classifier separates classes by a weighted score's sign
d(x)=sign⁡(g(x)),g(x)=wTx+b.d(\mathbf{x})=\operatorname{sign}(g(\mathbf{x})), \qquad g(\mathbf{x})=\mathbf{w}^T\mathbf{x}+b.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which side of the decision boundary contains input x?

Key idea

Calculate wᵀx+b: positive assigns one class, negative the other, and zero lies on the boundary.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Multiply corresponding features and weights

    wixiw_ix_i
  2. 2

    Sum the products and add bias

    g(x)=wTx+bg(x)=w^Tx+b
  3. 3

    Check the sign

    g(x)>0⇒+1,g(x)<0⇒−1g(x)>0\Rightarrow+1,\quad g(x)<0\Rightarrow-1
  4. 4

    Calculate boundary distance if needed

    distance=∣g(x)∣/∥w∥2\mathrm{distance}=|g(x)|/\lVert w\rVert_2

Worked example

Work through the numbers — Classifying one point

w=[1,-2], x=[3,1], and b=0.5.

  1. 1

    Weighted sum

    1⋅3+(−2)⋅1+0.5=1.51\cdot3+(-2)\cdot1+0.5=1.5
  2. 2

    Class

    sign⁡(1.5)=+1\operatorname{sign}(1.5)=+1
  3. 3

    Distance to the boundary

    1.5/12+(−2)2≈0.671.5/\sqrt{1^2+(-2)^2}\approx0.67

Result: The point is on the positive-class side, approximately 0.67 from the boundary.

Sanity check

Checks
  • w and x must have the same dimension.
  • Geometric distance requires |g(x)|, while signed distance may retain its sign.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
w\mathbf{w}Learned weight vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI5-007 Identifier: MAI5-007

Part I · Study equation

A hyperplane classifier separates classes by a weighted score's sign
sign⁡(a)={−1,a<0,0,a=0,1,a>0.\operatorname{sign}(a)= \begin{cases} -1,&a<0,\\ 0,&a=0,\\ 1,&a>0. \end{cases}
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which side of the decision boundary contains input x?

Key idea

Calculate wᵀx+b: positive assigns one class, negative the other, and zero lies on the boundary.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Multiply corresponding features and weights

    wixiw_ix_i
  2. 2

    Sum the products and add bias

    g(x)=wTx+bg(x)=w^Tx+b
  3. 3

    Check the sign

    g(x)>0⇒+1,g(x)<0⇒−1g(x)>0\Rightarrow+1,\quad g(x)<0\Rightarrow-1
  4. 4

    Calculate boundary distance if needed

    distance=∣g(x)∣/∥w∥2\mathrm{distance}=|g(x)|/\lVert w\rVert_2

Worked example

Work through the numbers — Classifying one point

w=[1,-2], x=[3,1], and b=0.5.

  1. 1

    Weighted sum

    1⋅3+(−2)⋅1+0.5=1.51\cdot3+(-2)\cdot1+0.5=1.5
  2. 2

    Class

    sign⁡(1.5)=+1\operatorname{sign}(1.5)=+1
  3. 3

    Distance to the boundary

    1.5/12+(−2)2≈0.671.5/\sqrt{1^2+(-2)^2}\approx0.67

Result: The point is on the positive-class side, approximately 0.67 from the boundary.

Sanity check

Checks
  • w and x must have the same dimension.
  • Geometric distance requires |g(x)|, while signed distance may retain its sign.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI5-008 Identifier: MAI5-008

#3.1 Decision-boundary geometry

  • The boundary is perpendicular to weight vector \mathbf{w}.
  • Bias shifts the boundary’s location.
  • The source’s b/\|w\|_2 can be read as signed distance. Nonnegative geometric distance is |b|/\|w\|_2.

Part I · Study equation

MSE has three steps: residuals, squares, and averaging
dist⁡(0,boundary)=b∥w∥2.\operatorname{dist}(0,\text{boundary})=\frac{b}{\|\mathbf{w}\|_2}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How far are predictions from targets on average?

Key idea

Subtract targets from predictions, square each residual, and divide by sample count. Larger errors receive greater penalties.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate predictions

    For linear regression, predictions are Xw or wᵀx.

    y^=Xw\hat y=Xw
  2. 2

    Calculate residuals

    r=y^−yr=\hat y-y
  3. 3

    Square the residuals

    Remove signs and emphasize larger errors.

    ri2r_i^2
  4. 4

    Sum and divide by sample count

    MSE=1L∑i=1Lri2\mathrm{MSE}=\frac{1}{L}\sum_{i=1}^{L}r_i^2

Worked example

Work through the numbers — MSE for two samples

Compare predictions and targets directly.

  1. 1

    Targets and predictions

    y=[3,1],y^=[2,4]y=[3,1],\quad\hat y=[2,4]
  2. 2

    Residual

    y^−y=[−1,3]\hat y-y=[-1,3]
  3. 3

    Square

    [−1,3]2=[1,9][-1,3]^2=[1,9]
  4. 4

    Average

    (1+9)/2=5(1+9)/2=5

Result: MSE is 5. The second sample's error of 3 substantially increases it.

Sanity check

Checks
  • Check whether the expression divides by sample count or is merely a sum of squares.
  • MSE has squared output units.

Symbols

Symbols used in this equation
SymbolMeaning
w\mathbf{w}Learned weight vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI5-009 Identifier: MAI5-009

Including the sign, a point x’s position relative to the boundary is expressed by:

Part I · Study equation

MSE has three steps: residuals, squares, and averaging
signed distance⁡(x)=g(x)∥w∥2.\operatorname{signed\ distance}(\mathbf{x}) = \frac{g(\mathbf{x})}{\|\mathbf{w}\|_2}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How far are predictions from targets on average?

Key idea

Subtract targets from predictions, square each residual, and divide by sample count. Larger errors receive greater penalties.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate predictions

    For linear regression, predictions are Xw or wᵀx.

    y^=Xw\hat y=Xw
  2. 2

    Calculate residuals

    r=y^−yr=\hat y-y
  3. 3

    Square the residuals

    Remove signs and emphasize larger errors.

    ri2r_i^2
  4. 4

    Sum and divide by sample count

    MSE=1L∑i=1Lri2\mathrm{MSE}=\frac{1}{L}\sum_{i=1}^{L}r_i^2

Worked example

Work through the numbers — MSE for two samples

Compare predictions and targets directly.

  1. 1

    Targets and predictions

    y=[3,1],y^=[2,4]y=[3,1],\quad\hat y=[2,4]
  2. 2

    Residual

    y^−y=[−1,3]\hat y-y=[-1,3]
  3. 3

    Square

    [−1,3]2=[1,9][-1,3]^2=[1,9]
  4. 4

    Average

    (1+9)/2=5(1+9)/2=5

Result: MSE is 5. The second sample's error of 3 substantially increases it.

Sanity check

Checks
  • Check whether the expression divides by sample count or is merely a sum of squares.
  • MSE has squared output units.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
w\mathbf{w}Learned weight vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI5-010 Identifier: MAI5-010

Part I · Study equation

Identify the equation's role and symbols
g(x)=0on the plane,g(x)>0on the positive side,g(x)<0on the negative side.\begin{aligned} g(\mathbf{x})&=0 &&\text{on the plane},\\ g(\mathbf{x})&>0 &&\text{on the positive side},\\ g(\mathbf{x})&<0 &&\text{on the negative side}. \end{aligned}
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI5-011 Identifier: MAI5-011

#4. Linearly separable training sets and zero error

Part I · Study equation

Identify the equation's role and symbols
D={(x1,y1),…,(xM,yM)}.\mathcal{D}= \{(\mathbf{x}_1,y_1),\ldots,(\mathbf{x}_M,y_M)\}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI5-012 Identifier: MAI5-012

Part I · Study equation

Identify the equation's role and symbols
(ym=1∧g(xm)>0) ∨ (ym=−1∧g(xm)<0),∀m.\big(y_m=1\land g(\mathbf{x}_m)>0\big) \ \lor\ \big(y_m=-1\land g(\mathbf{x}_m)<0\big), \qquad\forall m.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI5-013 Identifier: MAI5-013

In one line:

Part I · Study equation

Identify the equation's role and symbols
ym g(xm)>0,∀m.y_m\,g(\mathbf{x}_m)>0,\qquad\forall m.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI5-014 Identifier: MAI5-014

#5. Nonlinear discriminants and convolution

Begin with continuous convolution, one of the DNN’s key building blocks.

Part I · Study equation

Identify the equation's role and symbols
(f∗g)(t)=∫−∞∞f(τ)g(t−τ) dτ.(f*g)(t) = \int_{-\infty}^{\infty} f(\tau)g(t-\tau)\,d\tau.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI5-015 Identifier: MAI5-015

These two discrete forms are equivalent.

Part I · Study equation

Convolution slides a kernel and repeats elementwise products and sums
(f∗g)[n]=∑m=−∞∞f[m] g[n−m].(f*g)[n] = \sum_{m=-\infty}^{\infty} f[m]\,g[n-m].
Explanation and calculations 5 calculation steps · Numerical example · Checks included

The question this equation answers

How do filter weights summarize each location's neighborhood?

Key idea

Flip and shift the kernel, then sum products of overlapping entries. CNN implementations commonly use unflipped cross-correlation.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check kernel orientation and boundary conditions

    First distinguish mathematical convolution from CNN-style cross-correlation.

  2. 2

    Align the kernel with an output position

    Stride and padding determine the overlapping input region.

  3. 3

    Multiply overlapping entries

    f[k]h[n−k]f[k]h[n-k]
  4. 4

    Sum the products

    g[n]=∑kf[k]h[n−k]g[n]=\sum_k f[k]h[n-k]
  5. 5

    Move to the next position and repeat

    In 2D, perform the same operation over horizontal and vertical indices.

Worked example

Work through the numbers — 1D full convolution

Use signal [1,2,3] and kernel [1,1].

  1. 1

    First position

    1⋅1=11\cdot1=1
  2. 2

    Second position

    1⋅1+2⋅1=31\cdot1+2\cdot1=3
  3. 3

    Third position

    2⋅1+3⋅1=52\cdot1+3\cdot1=5
  4. 4

    Last position

    3⋅1=33\cdot1=3

Result: The full output is [1,3,5,3].

Note: Same or valid padding changes output length.

Sanity check

Checks
  • Output size depends on kernel, padding, stride, and dilation.
  • Keep kernel-flipping conventions consistent between code and equations.

Symbols

Symbols used in this equation
SymbolMeaning
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
CS231n · Convolutional Networks ↗ Visualize elementwise filter–patch products, sums, and output-size calculations. Deep Learning Book · Linear Algebra ↗ Reinterpret convolution through inner products and matrix multiplication.
Modern Artificial Intelligence I · MAI5-016 Identifier: MAI5-016

Part I · Study equation

Convolution slides a kernel and repeats elementwise products and sums
(f∗g)[n]=∑m=−∞∞f[n−m] g[m].(f*g)[n] = \sum_{m=-\infty}^{\infty} f[n-m]\,g[m].
Explanation and calculations 5 calculation steps · Numerical example · Checks included

The question this equation answers

How do filter weights summarize each location's neighborhood?

Key idea

Flip and shift the kernel, then sum products of overlapping entries. CNN implementations commonly use unflipped cross-correlation.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check kernel orientation and boundary conditions

    First distinguish mathematical convolution from CNN-style cross-correlation.

  2. 2

    Align the kernel with an output position

    Stride and padding determine the overlapping input region.

  3. 3

    Multiply overlapping entries

    f[k]h[n−k]f[k]h[n-k]
  4. 4

    Sum the products

    g[n]=∑kf[k]h[n−k]g[n]=\sum_k f[k]h[n-k]
  5. 5

    Move to the next position and repeat

    In 2D, perform the same operation over horizontal and vertical indices.

Worked example

Work through the numbers — 1D full convolution

Use signal [1,2,3] and kernel [1,1].

  1. 1

    First position

    1⋅1=11\cdot1=1
  2. 2

    Second position

    1⋅1+2⋅1=31\cdot1+2\cdot1=3
  3. 3

    Third position

    2⋅1+3⋅1=52\cdot1+3\cdot1=5
  4. 4

    Last position

    3⋅1=33\cdot1=3

Result: The full output is [1,3,5,3].

Note: Same or valid padding changes output length.

Sanity check

Checks
  • Output size depends on kernel, padding, stride, and dilation.
  • Keep kernel-flipping conventions consistent between code and equations.

Symbols

Symbols used in this equation
SymbolMeaning
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
CS231n · Convolutional Networks ↗ Visualize elementwise filter–patch products, sums, and output-size calculations. Deep Learning Book · Linear Algebra ↗ Reinterpret convolution through inner products and matrix multiplication.
Modern Artificial Intelligence I · MAI5-017 Identifier: MAI5-017

#6. Common DNN modules

The lecture’s PyTorch module list is reorganized by function below.

ModuleRole
nn.LinearFully connected linear transformation
nn.Conv2dTwo-dimensional convolution
nn.ConvTranspose2dTransposed-convolution upsampling
nn.ReLUNonlinear activation
nn.MaxPool2dMax pooling
nn.AvgPool2dAverage pooling
nn.UpsampleInterpolation-based upsampling
nn.DropoutRegularization through random deactivation during training

#7. 2026 update: what followed CNNs?

CNN mathematics has not become obsolete, but visual-model choices have broadened substantially.

  • Vision Transformer treats images as patch sequences and demonstrated strong image recognition with attention alone.
  • PyTorch 2.13 provides scaled_dot_product_attention and can dispatch to several backends, including FlashAttention-2-family implementations.
  • torch.compile is a central PyTorch execution path capturing model code as graphs for optimization.
  • CNNs remain widely used for low-level features, embedded vision, restoration, medical imaging, and hybrid architectures.

The lecture’s perceptron → convolution → DNN progression is a historical foundation, extended by attention and foundation models.

#Part 1.5: Random Variables, Conditional Probability, and Conditional Expectation

#1. CDF and PDF of a real random variable

Part I · Study equation

Identify the equation's role and symbols
FX(x)=P{X≤x},∀x∈R.F_X(x)=P\{X\le x\},\qquad \forall x\in\mathbb{R}.
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

From PMFs to expectation, variance, and conditional laws

Use a small discrete distribution and joint table to connect normalization, expectation, variance, conditional distributions, and conditional expectation.

Structure explanation
Loading the lesson matched to this equation and derivation step.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI6-001 Identifier: MAI6-001

Part I · Study equation

Identify the equation's role and symbols
lim⁡t→∞FX(t)=1,lim⁡t→−∞FX(t)=0.\lim_{t\to\infty}F_X(t)=1,\qquad \lim_{t\to-\infty}F_X(t)=0.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI6-002 Identifier: MAI6-002

Part I · Study equation

Identify the equation's role and symbols
pX(x)=dFX(x)dx.p_X(x)=\frac{dF_X(x)}{dx}.
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

From PMFs to expectation, variance, and conditional laws

Use a small discrete distribution and joint table to connect normalization, expectation, variance, conditional distributions, and conditional expectation.

Synchronized derivation
Loading the lesson matched to this equation and derivation step.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI6-003 Identifier: MAI6-003

#2. Discrete and continuous distributions

The lecture’s discrete examples are Bernoulli, binomial, and Poisson; continuous examples are uniform, Gaussian, and Laplace.

The Poisson PMF is:

Part I · Study equation

Identify the equation's role and symbols
P{X=k}=e−λλkk!,k=0,1,2,…,λ>0.P\{X=k\}=e^{-\lambda}\frac{\lambda^k}{k!}, \qquad k=0,1,2,\ldots,\quad \lambda>0.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
λ\lambdaRegularization strength or eigenvalue

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI6-004 Identifier: MAI6-004

A univariate Gaussian PDF with mean μ and variance σ² is:

Part I · Study equation

Variance averages squared deviations from the mean
pX(x)=12πσ2exp⁡ ⁣(−12(x−μ)2σ2).p_X(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left( -\frac12\frac{(x-\mu)^2}{\sigma^2} \right).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How widely does a random variable spread around its mean?

Key idea

Subtract the mean from each value, square, and take the probability-weighted average.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the mean

    μ=E[X]\mu=E[X]
  2. 2

    Calculate deviations from the mean

    X−μX-\mu
  3. 3

    Square the deviations

    (X−μ)2(X-\mu)^2
  4. 4

    Take the expectation

    Var⁡(X)=E[(X−μ)2]\operatorname{Var}(X)=E[(X-\mu)^2]

Worked example

Work through the numbers — X∈{1,2,3} with probabilities {0.2,0.5,0.3}

Use mean μ=2.1 from the preceding example.

  1. 1

    Squared deviations

    (1−2.1)2=1.21, (2−2.1)2=0.01, (3−2.1)2=0.81(1-2.1)^2=1.21,\ (2-2.1)^2=0.01,\ (3-2.1)^2=0.81
  2. 2

    Probability-weighted sum

    0.2(1.21)+0.5(0.01)+0.3(0.81)=0.490.2(1.21)+0.5(0.01)+0.3(0.81)=0.49

Result: Variance is 0.49 and standard deviation is √0.49=0.7.

Sanity check

Checks
  • Variance cannot be negative.
  • Standard deviation shares the variable's units; variance has squared units.

Symbols

Symbols used in this equation
SymbolMeaning
μ\muAverage
σ\sigmaStandard deviation or sigmoid notation

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-005 Identifier: MAI6-005

#3. Independence

Part I · Study equation

Identify the equation's role and symbols
pX,Y(x,y)=pX(x)pY(y).p_{X,Y}(x,y)=p_X(x)p_Y(y).
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

From PMFs to expectation, variance, and conditional laws

Use a small discrete distribution and joint table to connect normalization, expectation, variance, conditional distributions, and conditional expectation.

Structure explanation
Loading the lesson matched to this equation and derivation step.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI6-006 Identifier: MAI6-006

#4. Mean, expectation, variance, and correlation

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
μX=E[X]=∫x pX(x) dx.\mu_X=E[X]=\int x\,p_X(x)\,dx.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Formula-specific lesson

From PMFs to expectation, variance, and conditional laws

Use a small discrete distribution and joint table to connect normalization, expectation, variance, conditional distributions, and conditional expectation.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
μ\muAverage
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-007 Identifier: MAI6-007

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[g(X)]=∫g(x)pX(x) dx.E[g(X)]=\int g(x)p_X(x)\,dx.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Formula-specific lesson

From PMFs to expectation, variance, and conditional laws

Use a small discrete distribution and joint table to connect normalization, expectation, variance, conditional distributions, and conditional expectation.

Synchronized derivation
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-008 Identifier: MAI6-008

Part I · Study equation

Variance averages squared deviations from the mean
Var⁡(X)=E ⁣[∣X−E[X]∣2]=E[∣X∣2]−∣E[X]∣2.\operatorname{Var}(X) = E\!\left[|X-E[X]|^2\right] = E[|X|^2]-|E[X]|^2.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How widely does a random variable spread around its mean?

Key idea

Subtract the mean from each value, square, and take the probability-weighted average.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the mean

    μ=E[X]\mu=E[X]
  2. 2

    Calculate deviations from the mean

    X−μX-\mu
  3. 3

    Square the deviations

    (X−μ)2(X-\mu)^2
  4. 4

    Take the expectation

    Var⁡(X)=E[(X−μ)2]\operatorname{Var}(X)=E[(X-\mu)^2]

Worked example

Work through the numbers — X∈{1,2,3} with probabilities {0.2,0.5,0.3}

Use mean μ=2.1 from the preceding example.

  1. 1

    Squared deviations

    (1−2.1)2=1.21, (2−2.1)2=0.01, (3−2.1)2=0.81(1-2.1)^2=1.21,\ (2-2.1)^2=0.01,\ (3-2.1)^2=0.81
  2. 2

    Probability-weighted sum

    0.2(1.21)+0.5(0.01)+0.3(0.81)=0.490.2(1.21)+0.5(0.01)+0.3(0.81)=0.49

Result: Variance is 0.49 and standard deviation is √0.49=0.7.

Sanity check

Checks
  • Variance cannot be negative.
  • Standard deviation shares the variable's units; variance has squared units.

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-009 Identifier: MAI6-009

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[XY∗]=∬xy∗pX,Y(x,y) dx dy.E[XY^*] = \iint xy^*p_{X,Y}(x,y)\,dx\,dy.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-010 Identifier: MAI6-010

Expanding real and imaginary parts of complex random variables makes the correlation integral four-dimensional.

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[XY∗]=\iiiint(a+ib)(c−id) pXr,Xi,Yr,Yi(a,b,c,d) da db dc dd.E[XY^*] = \iiiint (a+ib)(c-id)\, p_{X_r,X_i,Y_r,Y_i}(a,b,c,d)\, da\,db\,dc\,dd.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-011 Identifier: MAI6-011

#5. Covariance

Part I · Study equation

Covariance measures how two variables deviate together from their means
Cov⁡(X,Y)=E[(X−E[X])(Y−E[Y])∗]=E[XY∗]−E[X]E[Y∗].\operatorname{Cov}(X,Y) = E[(X-E[X])(Y-E[Y])^*] = E[XY^*]-E[X]E[Y^*].
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

When X exceeds its mean, does Y tend to increase too?

Key idea

Multiply centered deviations and average. Positive indicates a same-direction tendency; negative indicates opposite directions.

Formula-specific lesson

From PMFs to expectation, variance, and conditional laws

Use a small discrete distribution and joint table to connect normalization, expectation, variance, conditional distributions, and conditional expectation.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate each variable's mean

    μX=E[X],μY=E[Y]\mu_X=E[X],\quad\mu_Y=E[Y]
  2. 2

    Calculate deviations

    X−μX,Y−μYX-\mu_X,\quad Y-\mu_Y
  3. 3

    Multiply paired deviations

    (X−μX)(Y−μY)(X-\mu_X)(Y-\mu_Y)
  4. 4

    Take an expectation or sample average

    Cov⁡(X,Y)=E[(X−μX)(Y−μY)]\operatorname{Cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)]

Worked example

Work through the numbers — Covariance direction with two samples

X=[1,3], Y=[2,6], with means 2 and 4.

  1. 1

    Deviations

    X−2=[−1,1],Y−4=[−2,2]X-2=[-1,1],\quad Y-4=[-2,2]
  2. 2

    Products of deviations

    [2,2][2,2]
  3. 3

    Average

    (2+2)/2=2(2+2)/2=2

Result: Positive covariance indicates the variables tend to increase together.

Sanity check

Checks
  • Cov(X,X)=Var(X).
  • Zero covariance does not generally guarantee independence.

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-012 Identifier: MAI6-012

Distinguish ensemble mean from a finite-sample mean.

Part I · Study equation

Probability calculations proceed through event definitions, conditions, and sum/product/normalization rules
xˉ=1N∑n=1Nxn.\bar{x}=\frac1N\sum_{n=1}^{N}x_n.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which event or observation interval's probability is required?

Key idea

Define the event, add mutually exclusive cases, and multiply chained conditions using conditional probabilities.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Translate events and conditions into words

    P(A|B) is the probability of A given B.

  2. 2

    Choose a sum rule or product rule

    Add mutually exclusive alternatives; multiply sequential conditions.

  3. 3

    Use total probability for the denominator if needed

  4. 4

    Check 0≤P≤1 and total probability 1

Worked example

Work through the numbers — Conditional probability

P(A∩B)=0.15 and P(B)=0.30.

  1. 1

    Apply the definition

    P(A∣B)=P(A∩B)P(B)P(A\mid B)=\frac{P(A\cap B)}{P(B)}
  2. 2

    Substitute values

    0.15/0.30=0.50.15/0.30=0.5

Result: Among cases where B occurs, A occurs half the time.

Sanity check

Checks
  • The conditioning event's probability in the denominator must be nonzero.
  • Distinguish density p(x) from event probability P(X∈A).

Symbols

Symbols used in this equation
SymbolMeaning
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-013 Identifier: MAI6-013

The covariance properties are listed in full.

Part I · Study equation

Covariance measures how two variables deviate together from their means
Cov⁡(X,Y)=E[XY∗]−μXμY∗.\operatorname{Cov}(X,Y) = E[XY^*]-\mu_X\mu_Y^*.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

When X exceeds its mean, does Y tend to increase too?

Key idea

Multiply centered deviations and average. Positive indicates a same-direction tendency; negative indicates opposite directions.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate each variable's mean

    μX=E[X],μY=E[Y]\mu_X=E[X],\quad\mu_Y=E[Y]
  2. 2

    Calculate deviations

    X−μX,Y−μYX-\mu_X,\quad Y-\mu_Y
  3. 3

    Multiply paired deviations

    (X−μX)(Y−μY)(X-\mu_X)(Y-\mu_Y)
  4. 4

    Take an expectation or sample average

    Cov⁡(X,Y)=E[(X−μX)(Y−μY)]\operatorname{Cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)]

Worked example

Work through the numbers — Covariance direction with two samples

X=[1,3], Y=[2,6], with means 2 and 4.

  1. 1

    Deviations

    X−2=[−1,1],Y−4=[−2,2]X-2=[-1,1],\quad Y-4=[-2,2]
  2. 2

    Products of deviations

    [2,2][2,2]
  3. 3

    Average

    (2+2)/2=2(2+2)/2=2

Result: Positive covariance indicates the variables tend to increase together.

Sanity check

Checks
  • Cov(X,X)=Var(X).
  • Zero covariance does not generally guarantee independence.

Symbols

Symbols used in this equation
SymbolMeaning
μ\muAverage
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-014 Identifier: MAI6-014

Part I · Study equation

Covariance measures how two variables deviate together from their means
Cov⁡(X,Y)=Cov⁡∗(Y,X).\operatorname{Cov}(X,Y) = \operatorname{Cov}^*(Y,X).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

When X exceeds its mean, does Y tend to increase too?

Key idea

Multiply centered deviations and average. Positive indicates a same-direction tendency; negative indicates opposite directions.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate each variable's mean

    μX=E[X],μY=E[Y]\mu_X=E[X],\quad\mu_Y=E[Y]
  2. 2

    Calculate deviations

    X−μX,Y−μYX-\mu_X,\quad Y-\mu_Y
  3. 3

    Multiply paired deviations

    (X−μX)(Y−μY)(X-\mu_X)(Y-\mu_Y)
  4. 4

    Take an expectation or sample average

    Cov⁡(X,Y)=E[(X−μX)(Y−μY)]\operatorname{Cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)]

Worked example

Work through the numbers — Covariance direction with two samples

X=[1,3], Y=[2,6], with means 2 and 4.

  1. 1

    Deviations

    X−2=[−1,1],Y−4=[−2,2]X-2=[-1,1],\quad Y-4=[-2,2]
  2. 2

    Products of deviations

    [2,2][2,2]
  3. 3

    Average

    (2+2)/2=2(2+2)/2=2

Result: Positive covariance indicates the variables tend to increase together.

Sanity check

Checks
  • Cov(X,X)=Var(X).
  • Zero covariance does not generally guarantee independence.

Symbols

Symbols used in this equation
SymbolMeaning
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-015 Identifier: MAI6-015

Part I · Study equation

Covariance measures how two variables deviate together from their means
Cov⁡(X,X)=Var⁡(X).\operatorname{Cov}(X,X)=\operatorname{Var}(X).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

When X exceeds its mean, does Y tend to increase too?

Key idea

Multiply centered deviations and average. Positive indicates a same-direction tendency; negative indicates opposite directions.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate each variable's mean

    μX=E[X],μY=E[Y]\mu_X=E[X],\quad\mu_Y=E[Y]
  2. 2

    Calculate deviations

    X−μX,Y−μYX-\mu_X,\quad Y-\mu_Y
  3. 3

    Multiply paired deviations

    (X−μX)(Y−μY)(X-\mu_X)(Y-\mu_Y)
  4. 4

    Take an expectation or sample average

    Cov⁡(X,Y)=E[(X−μX)(Y−μY)]\operatorname{Cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)]

Worked example

Work through the numbers — Covariance direction with two samples

X=[1,3], Y=[2,6], with means 2 and 4.

  1. 1

    Deviations

    X−2=[−1,1],Y−4=[−2,2]X-2=[-1,1],\quad Y-4=[-2,2]
  2. 2

    Products of deviations

    [2,2][2,2]
  3. 3

    Average

    (2+2)/2=2(2+2)/2=2

Result: Positive covariance indicates the variables tend to increase together.

Sanity check

Checks
  • Cov(X,X)=Var(X).
  • Zero covariance does not generally guarantee independence.

Symbols

Symbols used in this equation
SymbolMeaning
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-016 Identifier: MAI6-016

Part I · Study equation

Covariance measures how two variables deviate together from their means
Cov⁡(aX+b,cY+d)=ac∗Cov⁡(X,Y).\operatorname{Cov}(aX+b,cY+d) = ac^*\operatorname{Cov}(X,Y).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

When X exceeds its mean, does Y tend to increase too?

Key idea

Multiply centered deviations and average. Positive indicates a same-direction tendency; negative indicates opposite directions.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate each variable's mean

    μX=E[X],μY=E[Y]\mu_X=E[X],\quad\mu_Y=E[Y]
  2. 2

    Calculate deviations

    X−μX,Y−μYX-\mu_X,\quad Y-\mu_Y
  3. 3

    Multiply paired deviations

    (X−μX)(Y−μY)(X-\mu_X)(Y-\mu_Y)
  4. 4

    Take an expectation or sample average

    Cov⁡(X,Y)=E[(X−μX)(Y−μY)]\operatorname{Cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)]

Worked example

Work through the numbers — Covariance direction with two samples

X=[1,3], Y=[2,6], with means 2 and 4.

  1. 1

    Deviations

    X−2=[−1,1],Y−4=[−2,2]X-2=[-1,1],\quad Y-4=[-2,2]
  2. 2

    Products of deviations

    [2,2][2,2]
  3. 3

    Average

    (2+2)/2=2(2+2)/2=2

Result: Positive covariance indicates the variables tend to increase together.

Sanity check

Checks
  • Cov(X,X)=Var(X).
  • Zero covariance does not generally guarantee independence.

Symbols

Symbols used in this equation
SymbolMeaning
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-017 Identifier: MAI6-017

Part I · Study equation

Covariance measures how two variables deviate together from their means
∣Cov⁡(X,Y)∣≤σXσY,σX2=Var⁡(X), σY2=Var⁡(Y).|\operatorname{Cov}(X,Y)|\le\sigma_X\sigma_Y, \qquad \sigma_X^2=\operatorname{Var}(X),\ \sigma_Y^2=\operatorname{Var}(Y).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

When X exceeds its mean, does Y tend to increase too?

Key idea

Multiply centered deviations and average. Positive indicates a same-direction tendency; negative indicates opposite directions.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate each variable's mean

    μX=E[X],μY=E[Y]\mu_X=E[X],\quad\mu_Y=E[Y]
  2. 2

    Calculate deviations

    X−μX,Y−μYX-\mu_X,\quad Y-\mu_Y
  3. 3

    Multiply paired deviations

    (X−μX)(Y−μY)(X-\mu_X)(Y-\mu_Y)
  4. 4

    Take an expectation or sample average

    Cov⁡(X,Y)=E[(X−μX)(Y−μY)]\operatorname{Cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)]

Worked example

Work through the numbers — Covariance direction with two samples

X=[1,3], Y=[2,6], with means 2 and 4.

  1. 1

    Deviations

    X−2=[−1,1],Y−4=[−2,2]X-2=[-1,1],\quad Y-4=[-2,2]
  2. 2

    Products of deviations

    [2,2][2,2]
  3. 3

    Average

    (2+2)/2=2(2+2)/2=2

Result: Positive covariance indicates the variables tend to increase together.

Sanity check

Checks
  • Cov(X,X)=Var(X).
  • Zero covariance does not generally guarantee independence.

Symbols

Symbols used in this equation
SymbolMeaning
σ\sigmaStandard deviation or sigmoid notation
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-018 Identifier: MAI6-018

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
∣E[XY∗]∣≤E[∣X∣2]E[∣Y∣2].|E[XY^*]| \le \sqrt{E[|X|^2]E[|Y|^2]}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-019 Identifier: MAI6-019

Part I · Study equation

Covariance measures how two variables deviate together from their means
X⊥Y⟹E[XY]=μXμY⟹Cov⁡(X,Y)=0.X\perp Y \quad\Longrightarrow\quad E[XY]=\mu_X\mu_Y \quad\Longrightarrow\quad \operatorname{Cov}(X,Y)=0.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

When X exceeds its mean, does Y tend to increase too?

Key idea

Multiply centered deviations and average. Positive indicates a same-direction tendency; negative indicates opposite directions.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate each variable's mean

    μX=E[X],μY=E[Y]\mu_X=E[X],\quad\mu_Y=E[Y]
  2. 2

    Calculate deviations

    X−μX,Y−μYX-\mu_X,\quad Y-\mu_Y
  3. 3

    Multiply paired deviations

    (X−μX)(Y−μY)(X-\mu_X)(Y-\mu_Y)
  4. 4

    Take an expectation or sample average

    Cov⁡(X,Y)=E[(X−μX)(Y−μY)]\operatorname{Cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)]

Worked example

Work through the numbers — Covariance direction with two samples

X=[1,3], Y=[2,6], with means 2 and 4.

  1. 1

    Deviations

    X−2=[−1,1],Y−4=[−2,2]X-2=[-1,1],\quad Y-4=[-2,2]
  2. 2

    Products of deviations

    [2,2][2,2]
  3. 3

    Average

    (2+2)/2=2(2+2)/2=2

Result: Positive covariance indicates the variables tend to increase together.

Sanity check

Checks
  • Cov(X,X)=Var(X).
  • Zero covariance does not generally guarantee independence.

Symbols

Symbols used in this equation
SymbolMeaning
μ\muAverage
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values
CCTotal class count or a matrix; check context
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-020 Identifier: MAI6-020

The converse does not generally hold. Jointly Gaussian variables are a special case where uncorrelatedness implies independence.

Part I · Study equation

Covariance measures how two variables deviate together from their means
Cov⁡ ⁣(∑nXn,∑mYm)=∑i∑jCov⁡(Xi,Yj).\operatorname{Cov}\!\left(\sum_n X_n,\sum_mY_m\right) = \sum_i\sum_j\operatorname{Cov}(X_i,Y_j).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

When X exceeds its mean, does Y tend to increase too?

Key idea

Multiply centered deviations and average. Positive indicates a same-direction tendency; negative indicates opposite directions.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate each variable's mean

    μX=E[X],μY=E[Y]\mu_X=E[X],\quad\mu_Y=E[Y]
  2. 2

    Calculate deviations

    X−μX,Y−μYX-\mu_X,\quad Y-\mu_Y
  3. 3

    Multiply paired deviations

    (X−μX)(Y−μY)(X-\mu_X)(Y-\mu_Y)
  4. 4

    Take an expectation or sample average

    Cov⁡(X,Y)=E[(X−μX)(Y−μY)]\operatorname{Cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)]

Worked example

Work through the numbers — Covariance direction with two samples

X=[1,3], Y=[2,6], with means 2 and 4.

  1. 1

    Deviations

    X−2=[−1,1],Y−4=[−2,2]X-2=[-1,1],\quad Y-4=[-2,2]
  2. 2

    Products of deviations

    [2,2][2,2]
  3. 3

    Average

    (2+2)/2=2(2+2)/2=2

Result: Positive covariance indicates the variables tend to increase together.

Sanity check

Checks
  • Cov(X,X)=Var(X).
  • Zero covariance does not generally guarantee independence.

Symbols

Symbols used in this equation
SymbolMeaning
∑\sumSum over all specified terms
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-021 Identifier: MAI6-021

#6. A vector proof of Cauchy–Schwarz

The lecture footnote’s derivation is retained too.

Part I · Study equation

Probability calculations proceed through event definitions, conditions, and sum/product/normalization rules
∑n=1N∑n′=1N(xnyn′−xn′yn)2=∑nxn2∑n′yn′2+∑nyn2∑n′xn′2−2∑nxnyn∑n′xn′yn′=2(∑nxn2)(∑nyn2)−2(∑nxnyn)2.\begin{aligned} \sum_{n=1}^{N}\sum_{n'=1}^{N} (x_ny_{n'}-x_{n'}y_n)^2 &= \sum_n x_n^2\sum_{n'}y_{n'}^2 + \sum_n y_n^2\sum_{n'}x_{n'}^2\\ &\quad- 2\sum_n x_ny_n\sum_{n'}x_{n'}y_{n'}\\ &= 2\left(\sum_nx_n^2\right) \left(\sum_ny_n^2\right) - 2\left(\sum_nx_ny_n\right)^2. \end{aligned}
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which event or observation interval's probability is required?

Key idea

Define the event, add mutually exclusive cases, and multiply chained conditions using conditional probabilities.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Translate events and conditions into words

    P(A|B) is the probability of A given B.

  2. 2

    Choose a sum rule or product rule

    Add mutually exclusive alternatives; multiply sequential conditions.

  3. 3

    Use total probability for the denominator if needed

  4. 4

    Check 0≤P≤1 and total probability 1

Worked example

Work through the numbers — Conditional probability

P(A∩B)=0.15 and P(B)=0.30.

  1. 1

    Apply the definition

    P(A∣B)=P(A∩B)P(B)P(A\mid B)=\frac{P(A\cap B)}{P(B)}
  2. 2

    Substitute values

    0.15/0.30=0.50.15/0.30=0.5

Result: Among cases where B occurs, A occurs half the time.

Sanity check

Checks
  • The conditioning event's probability in the denominator must be nonzero.
  • Distinguish density p(x) from event probability P(X∈A).

Symbols

Symbols used in this equation
SymbolMeaning
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-022 Identifier: MAI6-022

Part I · Study equation

Probability calculations proceed through event definitions, conditions, and sum/product/normalization rules
(∑n=1Nxn2)(∑n=1Nyn2)≥(∑n=1Nxnyn)2.\left(\sum_{n=1}^{N}x_n^2\right) \left(\sum_{n=1}^{N}y_n^2\right) \ge \left(\sum_{n=1}^{N}x_ny_n\right)^2.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which event or observation interval's probability is required?

Key idea

Define the event, add mutually exclusive cases, and multiply chained conditions using conditional probabilities.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Translate events and conditions into words

    P(A|B) is the probability of A given B.

  2. 2

    Choose a sum rule or product rule

    Add mutually exclusive alternatives; multiply sequential conditions.

  3. 3

    Use total probability for the denominator if needed

  4. 4

    Check 0≤P≤1 and total probability 1

Worked example

Work through the numbers — Conditional probability

P(A∩B)=0.15 and P(B)=0.30.

  1. 1

    Apply the definition

    P(A∣B)=P(A∩B)P(B)P(A\mid B)=\frac{P(A\cap B)}{P(B)}
  2. 2

    Substitute values

    0.15/0.30=0.50.15/0.30=0.5

Result: Among cases where B occurs, A occurs half the time.

Sanity check

Checks
  • The conditioning event's probability in the denominator must be nonzero.
  • Distinguish density p(x) from event probability P(X∈A).

Symbols

Symbols used in this equation
SymbolMeaning
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-023 Identifier: MAI6-023

#7. Conditional probability

Part I · Study equation

Probability calculations proceed through event definitions, conditions, and sum/product/normalization rules
P(A∣B).P(A\mid B).
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

Which event or observation interval's probability is required?

Key idea

Define the event, add mutually exclusive cases, and multiply chained conditions using conditional probabilities.

Formula-specific lesson

From PMFs to expectation, variance, and conditional laws

Use a small discrete distribution and joint table to connect normalization, expectation, variance, conditional distributions, and conditional expectation.

Structure explanation
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Translate events and conditions into words

    P(A|B) is the probability of A given B.

  2. 2

    Choose a sum rule or product rule

    Add mutually exclusive alternatives; multiply sequential conditions.

  3. 3

    Use total probability for the denominator if needed

  4. 4

    Check 0≤P≤1 and total probability 1

Worked example

Work through the numbers — Conditional probability

P(A∩B)=0.15 and P(B)=0.30.

  1. 1

    Apply the definition

    P(A∣B)=P(A∩B)P(B)P(A\mid B)=\frac{P(A\cap B)}{P(B)}
  2. 2

    Substitute values

    0.15/0.30=0.50.15/0.30=0.5

Result: Among cases where B occurs, A occurs half the time.

Sanity check

Checks
  • The conditioning event's probability in the denominator must be nonzero.
  • Distinguish density p(x) from event probability P(X∈A).

Symbols

Symbols used in this equation
SymbolMeaning
P(⋅)P(\cdot)Event probability

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-024 Identifier: MAI6-024

Part I · Study equation

Identify the equation's role and symbols
FY(y)=lim⁡x→∞FX,Y(x,y).F_Y(y)=\lim_{x\to\infty}F_{X,Y}(x,y).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI6-025 Identifier: MAI6-025

Part I · Study equation

Identify the equation's role and symbols
FX,Y(x,y)=∫−∞yFX∣Y(x∣t) dFY(t).F_{X,Y}(x,y) = \int_{-\infty}^{y} F_{X\mid Y}(x\mid t)\,dF_Y(t).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI6-026 Identifier: MAI6-026

Part I · Study equation

Identify the equation's role and symbols
FX∣Y(x∣y)=dFX,Y(x,y)dy(dFX,Y(∞,y)dy)−1.F_{X\mid Y}(x\mid y) = \frac{dF_{X,Y}(x,y)}{dy} \left( \frac{dF_{X,Y}(\infty,y)}{dy} \right)^{-1}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI6-027 Identifier: MAI6-027

The density-function form may be more familiar.

Part I · Study equation

Identify the equation's role and symbols
pX,Y(x,y)=d2FX,Y(x,y)dx dy.p_{X,Y}(x,y) = \frac{d^2F_{X,Y}(x,y)}{dx\,dy}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI6-028 Identifier: MAI6-028

Part I · Study equation

Identify the equation's role and symbols
pY(y)=dF(∞,y)dy.p_Y(y)=\frac{dF(\infty,y)}{dy}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI6-029 Identifier: MAI6-029

Part I · Study equation

Identify the equation's role and symbols
pX∣Y(x∣y)=pX,Y(x,y)pY(y).p_{X\mid Y}(x\mid y) = \frac{p_{X,Y}(x,y)}{p_Y(y)}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI6-030 Identifier: MAI6-030

Under independence, conditioning does not change X’s distribution.

Part I · Study equation

Identify the equation's role and symbols
pX∣Y(x∣y)=pX(x),∀x,y with pY(y)>0.p_{X\mid Y}(x\mid y)=p_X(x), \qquad \forall x,y\ \text{with }p_Y(y)>0.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI6-031 Identifier: MAI6-031

#8. Conditional expectation

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[X∣Y]=∫−∞∞x dFX∣Y(x∣Y).E[X\mid Y] = \int_{-\infty}^{\infty} x\,dF_{X\mid Y}(x\mid Y).
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Formula-specific lesson

From PMFs to expectation, variance, and conditional laws

Use a small discrete distribution and joint table to connect normalization, expectation, variance, conditional distributions, and conditional expectation.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-032 Identifier: MAI6-032

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[X∣Y]=∫−∞∞x pX∣Y(x∣Y) dx.E[X\mid Y] = \int_{-\infty}^{\infty} x\,p_{X\mid Y}(x\mid Y)\,dx.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Formula-specific lesson

From PMFs to expectation, variance, and conditional laws

Use a small discrete distribution and joint table to connect normalization, expectation, variance, conditional distributions, and conditional expectation.

Synchronized derivation
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-033 Identifier: MAI6-033

Conditional expectation is a function of Y and therefore itself a random variable.

Part I · Study equation

Identify the equation's role and symbols
IA(X)={1,X∈A,0,otherwise.I_A(X)= \begin{cases} 1,&X\in A,\\ 0,&\text{otherwise}. \end{cases}
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI6-034 Identifier: MAI6-034

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
P{X∈A∣Y}=E[IA(X)∣Y].P\{X\in A\mid Y\}=E[I_A(X)\mid Y].
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-035 Identifier: MAI6-035

#8.1 Filtration property

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[E[X∣Y,Z]∣Y]=E[X∣Y].E[E[X\mid Y,Z]\mid Y]=E[X\mid Y].
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-036 Identifier: MAI6-036

The integral expansion of the filtration property in the lecture footnote is also retained.

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[X∣Y=y]=∫x pX∣Y(x∣y) dx=∫x pX,Y(x,y)pY(y) dx=∫x ∫pX,Y,Z(x,y,z) dzpY(y) dx=∫pY,Z(y,z)pY(y){∫x pX∣Y,Z(x∣y,z) dx}dz=∫pZ∣Y(z∣y) E[X∣Y=y,Z=z] dz=E[E[X∣Y,Z]∣Y=y].\begin{aligned} E[X\mid Y=y] &=\int x\,p_{X\mid Y}(x\mid y)\,dx\\ &=\int x\,\frac{p_{X,Y}(x,y)}{p_Y(y)}\,dx\\ &=\int x\,\frac{\int p_{X,Y,Z}(x,y,z)\,dz}{p_Y(y)}\,dx\\ &=\int \frac{p_{Y,Z}(y,z)}{p_Y(y)} \left\{\int x\,p_{X\mid Y,Z}(x\mid y,z)\,dx\right\}dz\\ &=\int p_{Z\mid Y}(z\mid y)\,E[X\mid Y=y,Z=z]\,dz\\ &=E[E[X\mid Y,Z]\mid Y=y]. \end{aligned}
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-036A Identifier: MAI6-036A

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[X]=E[E[X∣Y]].E[X]=E[E[X\mid Y]].
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-037 Identifier: MAI6-037

#8.2 Conditioning on an already known quantity

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[f(X)Z∣X,Y]=f(X)E[Z∣X,Y].E[f(X)Z\mid X,Y] = f(X)E[Z\mid X,Y].
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-038 Identifier: MAI6-038

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[X∣X,Y]=X.E[X\mid X,Y]=X.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-039 Identifier: MAI6-039

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[E[X∣Y,Z]∣Y]=E[X∣Y]=E[E[X∣Y]∣Y,Z].E[E[X\mid Y,Z]\mid Y] = E[X\mid Y] = E[E[X\mid Y]\mid Y,Z].
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-040 Identifier: MAI6-040

#9. Deterministic or random?

The lecture’s check questions are retained as equations too.

Part I · Study equation

Identify the equation's role and symbols
Y=f(X).Y=f(X).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI6-041 Identifier: MAI6-041

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
μ=E[X].\mu=E[X].
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
μ\muAverage
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-042 Identifier: MAI6-042

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
X^=E[X∣Z].\widehat{X}=E[X\mid Z].
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-043 Identifier: MAI6-043

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
X^=E[X∣Z=z0].\widehat{X}=E[X\mid Z=z_0].
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI6-044 Identifier: MAI6-044

Interpret them as follows.

  • Y=f(X): generally a random variable
  • \mu=E[X]: a deterministic scalar once its distribution is fixed
  • E[X\mid Z]: a random variable because it is a function of Z
  • E[X\mid Z=z_0]: a deterministic quantity when conditioned on a particular z₀

#10. Implementation perspective in 2026

As of the source’s August 2026 update, PyTorch 2.13’s torch.distributions treats Bernoulli, normal, and Poisson distributions as more than sampling tools. Through log_prob, entropy, and reparameterized sampling, it connects probabilistic models to computation graphs.

Conditional probability and expectation are therefore both classical statistical language and foundations for implementing VAEs, stochastic policies, Bayesian layers, and uncertainty estimation.

#Part 1.6: Random Vectors, Joint Gaussians, WSS, and Power Spectra

#1. Random vectors

Part I · Study equation

Identify the equation's role and symbols
X=(X1,…,XN)∈X,X=RN or CN.\mathbf{X}=(X_1,\ldots,X_N)\in\mathcal{X}, \qquad \mathcal{X}=\mathbb{R}^N\ \text{or}\ \mathbb{C}^N.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
X\mathbf{X}Data matrix collecting samples
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-001 Identifier: MAI7-001

Part I · Study equation

Identify the equation's role and symbols
FX(x1,…,xN)=P{X1≤x1,…,XN≤xN}.F_{\mathbf{X}}(x_1,\ldots,x_N) = P\{X_1\le x_1,\ldots,X_N\le x_N\}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
X\mathbf{X}Data matrix collecting samples

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-002 Identifier: MAI7-002

Part I · Study equation

Identify the equation's role and symbols
lim⁡tn→∞FX(t1,…,tN)=1,lim⁡tn→−∞FX(t1,…,tN)=0.\lim_{t_n\to\infty} F_{\mathbf{X}}(t_1,\ldots,t_N)=1,\qquad \lim_{t_n\to-\infty} F_{\mathbf{X}}(t_1,\ldots,t_N)=0.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
X\mathbf{X}Data matrix collecting samples

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-003 Identifier: MAI7-003

Part I · Study equation

A gradient describes the direction and rate of loss change
pX(x)=∂NFX(x1,…,xN)∂x1⋯∂xN.p_{\mathbf{X}}(\mathbf{x}) = \frac{\partial^N F_{\mathbf{X}}(x_1,\ldots,x_N)} {\partial x_1\cdots\partial x_N}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
X\mathbf{X}Data matrix collecting samples
∂\partialPartial derivative with respect to a particular variable

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI7-004 Identifier: MAI7-004

#2. Multivariate Gaussian distributions

Part I · Study equation

Gaussian density uses standardized distance from the mean
X∼N(μ,C).\mathbf{X}\sim\mathcal{N}(\boldsymbol{\mu},C).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How plausible is observation x under mean μ and covariance Σ?

Key idea

Calculate a covariance-scaled distance from the mean and a normalization constant to obtain density.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the difference from the mean

    r=x−μr=x-\mu
  2. 2

    Adjust distance using variance or covariance

    d2=rTΣ−1rd^2=r^T\Sigma^{-1}r
  3. 3

    Calculate exponential decay

    exp⁡(−d2/2)\exp(-d^2/2)
  4. 4

    Divide by the normalization constant

    (2π)N/2∣Σ∣1/2(2\pi)^{N/2}|\Sigma|^{1/2}

Worked example

Work through the numbers — x=1 under a standard normal

A one-dimensional Gaussian with μ=0 and σ=1.

  1. 1

    Standardized distance

    z=(1−0)/1=1z=(1-0)/1=1
  2. 2

    Exponential term

    e−12/2=e−0.5≈0.6065e^{-1^2/2}=e^{-0.5}\approx0.6065
  3. 3

    Normalization

    0.6065/2π≈0.24200.6065/\sqrt{2\pi}\approx0.2420

Result: Density at x=1 is approximately 0.242. A density value is not itself an event probability.

Sanity check

Checks
  • Σ must be symmetric positive definite for the inverse and positive density to be stably defined.
  • Obtain probabilities through interval integration, distinct from pointwise density.

Symbols

Symbols used in this equation
SymbolMeaning
X\mathbf{X}Data matrix collecting samples
μ\muAverage
N\mathcal{N}Gaussian distribution
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-005 Identifier: MAI7-005

Part I · Study equation

Identify the equation's role and symbols
pX(x)=1det⁡(2πC)exp⁡ ⁣(−12(x−μ)HC−1(x−μ)).p_{\mathbf{X}}(\mathbf{x}) = \frac{1}{\sqrt{\det(2\pi C)}} \exp\!\left( -\frac12 (\mathbf{x}-\boldsymbol{\mu})^H C^{-1} (\mathbf{x}-\boldsymbol{\mu}) \right).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
X\mathbf{X}Data matrix collecting samples
μ\muAverage
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-006 Identifier: MAI7-006

Part I · Study equation

Identify the equation's role and symbols
det⁡(αC)=αNdet⁡(C),C∈CN×N.\det(\alpha C)=\alpha^N\det(C), \qquad C\in\mathbb{C}^{N\times N}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
α\alphaLearning rate, mixture coefficient, or dual coefficient
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-007 Identifier: MAI7-007

#3. Expectation, correlation, and covariance of random vectors

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[X]=∫Xx pX(x) dx.E[\mathbf{X}] = \int_{\mathcal{X}}\mathbf{x}\,p_{\mathbf{X}}(\mathbf{x})\,d\mathbf{x}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
X\mathbf{X}Data matrix collecting samples
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-008 Identifier: MAI7-008

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[g(X)]=∫Xg(x)pX(x) dx.E[g(\mathbf{X})] = \int_{\mathcal{X}}g(\mathbf{x})p_{\mathbf{X}}(\mathbf{x})\,d\mathbf{x}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
X\mathbf{X}Data matrix collecting samples
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-009 Identifier: MAI7-009

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
RX=E[XXH],[RX]i,j=E[XiXj∗]=∫Xxixj∗pX(x) dx.R_{\mathbf{X}} = E[\mathbf{X}\mathbf{X}^H], \qquad [R_{\mathbf{X}}]_{i,j} = E[X_iX_j^*] = \int_{\mathcal{X}} x_i x_j^* p_{\mathbf{X}}(\mathbf{x})\,d\mathbf{x}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
X\mathbf{X}Data matrix collecting samples
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-010 Identifier: MAI7-010

Part I · Study equation

Covariance measures how two variables deviate together from their means
CX,Y=E[(X−E[X])(Y−E[Y])H],[CX,Y]i,j=Cov⁡(Xi,Yj).C_{\mathbf{X},\mathbf{Y}} = E[(\mathbf{X}-E[\mathbf{X}]) (\mathbf{Y}-E[\mathbf{Y}])^H], \qquad [C_{\mathbf{X},\mathbf{Y}}]_{i,j} = \operatorname{Cov}(X_i,Y_j).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

When X exceeds its mean, does Y tend to increase too?

Key idea

Multiply centered deviations and average. Positive indicates a same-direction tendency; negative indicates opposite directions.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate each variable's mean

    μX=E[X],μY=E[Y]\mu_X=E[X],\quad\mu_Y=E[Y]
  2. 2

    Calculate deviations

    X−μX,Y−μYX-\mu_X,\quad Y-\mu_Y
  3. 3

    Multiply paired deviations

    (X−μX)(Y−μY)(X-\mu_X)(Y-\mu_Y)
  4. 4

    Take an expectation or sample average

    Cov⁡(X,Y)=E[(X−μX)(Y−μY)]\operatorname{Cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)]

Worked example

Work through the numbers — Covariance direction with two samples

X=[1,3], Y=[2,6], with means 2 and 4.

  1. 1

    Deviations

    X−2=[−1,1],Y−4=[−2,2]X-2=[-1,1],\quad Y-4=[-2,2]
  2. 2

    Products of deviations

    [2,2][2,2]
  3. 3

    Average

    (2+2)/2=2(2+2)/2=2

Result: Positive covariance indicates the variables tend to increase together.

Sanity check

Checks
  • Cov(X,X)=Var(X).
  • Zero covariance does not generally guarantee independence.

Symbols

Symbols used in this equation
SymbolMeaning
X\mathbf{X}Data matrix collecting samples
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-011 Identifier: MAI7-011

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
CX,X=RX−E[X]E[X]H.C_{\mathbf{X},\mathbf{X}} = R_{\mathbf{X}} - E[\mathbf{X}]E[\mathbf{X}]^H.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Formula-specific lesson

Quadratic forms, eigenvalues, covariance, and PCA axes

Use one symmetric matrix and a 2D sample cloud to calculate quadratic forms, eigenpairs, trace/determinant identities, covariance symmetry, and principal axes together.

Synchronized derivation
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
X\mathbf{X}Data matrix collecting samples
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-012 Identifier: MAI7-012

Independent components yield a diagonal covariance matrix.

Part I · Study equation

Identify the equation's role and symbols
CX,X=diag⁡(σ12,…,σN2).C_{\mathbf{X},\mathbf{X}} = \operatorname{diag}(\sigma_1^2,\ldots,\sigma_N^2).
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

Quadratic forms, eigenvalues, covariance, and PCA axes

Use one symmetric matrix and a 2D sample cloud to calculate quadratic forms, eigenpairs, trace/determinant identities, covariance symmetry, and principal axes together.

Structure explanation
Loading the lesson matched to this equation and derivation step.

Symbols

Symbols used in this equation
SymbolMeaning
X\mathbf{X}Data matrix collecting samples
σ\sigmaStandard deviation or sigmoid notation
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-013 Identifier: MAI7-013

The quadratic form of a positive-definite matrix appears again.

Part I · Study equation

A norm summarizes many components as one magnitude
∥x∥A2=xHAx>0,∀x∈CN, x≠0.\|\mathbf{x}\|_A^2 = \mathbf{x}^HA\mathbf{x}>0, \qquad \forall\mathbf{x}\in\mathbb{C}^N,\ \mathbf{x}\neq0.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How far is a vector or error from the origin?

Key idea

The L2 norm sums squared components and takes the square root. Squared norm omits the final square root.

Formula-specific lesson

Quadratic forms, eigenvalues, covariance, and PCA axes

Use one symmetric matrix and a 2D sample cloud to calculate quadratic forms, eigenpairs, trace/determinant identities, covariance symmetry, and principal axes together.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Square each component

    Turn both negative and positive components into nonnegative contributions.

  2. 2

    Sum the squares

    ∑ixi2\sum_i x_i^2
  3. 3

    Take the square root for a norm

    ∥x∥2=∑ixi2\lVert x\rVert_2=\sqrt{\sum_i x_i^2}
  4. 4

    Distinguish the squared norm

    ∥x∥22=∑ixi2\lVert x\rVert_2^2=\sum_i x_i^2

Worked example

Work through the numbers — The 3–4–5 vector

A familiar Pythagorean example.

  1. 1

    Vector

    x=[3,4]Tx=[3,4]^T
  2. 2

    Sum of squares

    32+42=9+16=253^2+4^2=9+16=25
  3. 3

    Square root

    25=5\sqrt{25}=5

Result: Thus ||x||₂=5 and ||x||₂²=25.

Sanity check

Checks
  • A norm is always nonnegative.
  • Confusing a norm with its square changes units and derivative coefficients.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI7-014 Identifier: MAI7-014

#4. Orthogonal covariance diagonalization and decorrelation

Part I · Study equation

Identify the equation's role and symbols
C=EΛEH.C=E\Lambda E^H.
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

Quadratic forms, eigenvalues, covariance, and PCA axes

Use one symmetric matrix and a 2D sample cloud to calculate quadratic forms, eigenpairs, trace/determinant identities, covariance symmetry, and principal axes together.

Interactive
Loading the lesson matched to this equation and derivation step.

Symbols

Symbols used in this equation
SymbolMeaning
CCTotal class count or a matrix; check context
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-015 Identifier: MAI7-015

Part I · Study equation

Identify the equation's role and symbols
X~=EHX.\widetilde{\mathbf{X}}=E^H\mathbf{X}.
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

Quadratic forms, eigenvalues, covariance, and PCA axes

Use one symmetric matrix and a 2D sample cloud to calculate quadratic forms, eigenpairs, trace/determinant identities, covariance symmetry, and principal axes together.

Interactive
Loading the lesson matched to this equation and derivation step.

Symbols

Symbols used in this equation
SymbolMeaning
X\mathbf{X}Data matrix collecting samples

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-016 Identifier: MAI7-016

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[X~X~H]=Λ.E[\widetilde{\mathbf{X}}\widetilde{\mathbf{X}}^H]=\Lambda.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
X\mathbf{X}Data matrix collecting samples
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-017 Identifier: MAI7-017

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[X~iX~j]=0,i≠j.E[\widetilde{X}_i\widetilde{X}_j]=0, \qquad i\neq j.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-018 Identifier: MAI7-018

For complex vectors, conjugation is ordinarily included as follows.

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[X~iX~j∗]=0,i≠j.E[\widetilde{X}_i\widetilde{X}_j^*]=0, \qquad i\neq j.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-018C Identifier: MAI7-018C

For jointly Gaussian variables, uncorrelated components are jointly independent.

#5. Conditional expectation for jointly Gaussian variables

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E[X∣Y]=AY+b.E[\mathbf{X}\mid\mathbf{Y}] = A\mathbf{Y}+\mathbf{b}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
X\mathbf{X}Data matrix collecting samples
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-019 Identifier: MAI7-019

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
E ⁣[(X−E[X∣Y])(X−E[X∣Y])H∣Y]=C.E\!\left[ (\mathbf{X}-E[\mathbf{X}\mid\mathbf{Y}]) (\mathbf{X}-E[\mathbf{X}\mid\mathbf{Y}])^H \mid\mathbf{Y} \right] = C.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
X\mathbf{X}Data matrix collecting samples
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-020 Identifier: MAI7-020

#6. An example with M i.i.d. Gaussian vectors

Part I · Study equation

Matrix multiplication repeats row–column inner products
X=[X1⋯XM].\mathbf{X}= \begin{bmatrix} \mathbf{X}_1&\cdots&\mathbf{X}_M \end{bmatrix}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which products and sums of inputs form each output entry?

Key idea

Multiply corresponding entries of one left-matrix row and one right-matrix column, then sum.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Check multiplication compatibility

    The inner dimensions must agree.

    (m×n)(n×p)→(m×p)(m\times n)(n\times p)\rightarrow(m\times p)
  2. 2

    Choose one output position

    Output entry (i,j) uses row i of A and column j of B.

  3. 3

    Multiply matching indices and sum

    Cij=∑kAikBkjC_{ij}=\sum_k A_{ik}B_{kj}
  4. 4

    Repeat for every row–column pair

    Apply the procedure across the entire output matrix.

Worked example

Work through the numbers — Multiplying a 2×2 matrix by a vector

Use numbers to see how each row transforms the vector.

  1. 1

    Input

    A=[1234],x=[56]A=\begin{bmatrix}1&2\\3&4\end{bmatrix},\quad x=\begin{bmatrix}5\\6\end{bmatrix}
  2. 2

    First row

    1⋅5+2⋅6=171\cdot5+2\cdot6=17
  3. 3

    Second row

    3⋅5+4⋅6=393\cdot5+4\cdot6=39

Result: The result is Ax=[17,39]^T.

Sanity check

Checks
  • Changing multiplication order generally changes the result.
  • Output shape is the outer dimensions m×p.

Symbols

Symbols used in this equation
SymbolMeaning
X\mathbf{X}Data matrix collecting samples

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI7-021 Identifier: MAI7-021

Part I · Study equation

Gaussian density uses standardized distance from the mean
pX(x)=∏m=1M1(2π)N/2∣C∣−1/2exp⁡ ⁣{−12(xm−μ)TC−1(xm−μ)}.p_{\mathbf{X}}(\mathbf{x}) = \prod_{m=1}^{M} \frac{1}{(2\pi)^{N/2}}|C|^{-1/2} \exp\!\left\{ -\frac12 (\mathbf{x}_m-\boldsymbol{\mu})^T C^{-1} (\mathbf{x}_m-\boldsymbol{\mu}) \right\}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How plausible is observation x under mean μ and covariance Σ?

Key idea

Calculate a covariance-scaled distance from the mean and a normalization constant to obtain density.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the difference from the mean

    r=x−μr=x-\mu
  2. 2

    Adjust distance using variance or covariance

    d2=rTΣ−1rd^2=r^T\Sigma^{-1}r
  3. 3

    Calculate exponential decay

    exp⁡(−d2/2)\exp(-d^2/2)
  4. 4

    Divide by the normalization constant

    (2π)N/2∣Σ∣1/2(2\pi)^{N/2}|\Sigma|^{1/2}

Worked example

Work through the numbers — x=1 under a standard normal

A one-dimensional Gaussian with μ=0 and σ=1.

  1. 1

    Standardized distance

    z=(1−0)/1=1z=(1-0)/1=1
  2. 2

    Exponential term

    e−12/2=e−0.5≈0.6065e^{-1^2/2}=e^{-0.5}\approx0.6065
  3. 3

    Normalization

    0.6065/2π≈0.24200.6065/\sqrt{2\pi}\approx0.2420

Result: Density at x=1 is approximately 0.242. A density value is not itself an event probability.

Sanity check

Checks
  • Σ must be symmetric positive definite for the inverse and positive density to be stably defined.
  • Obtain probabilities through interval integration, distinct from pointwise density.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
X\mathbf{X}Data matrix collecting samples
μ\muAverage
∏\prodMultiply all specified terms
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-022 Identifier: MAI7-022

Part I · Study equation

Gaussian density uses standardized distance from the mean
pX(x)=1(2π)MN/2∣C∣−1/2exp⁡ ⁣{−12∑m=1M(xm−μ)TC−1(xm−μ)}.p_{\mathbf{X}}(\mathbf{x}) = \frac{1}{(2\pi)^{MN/2}}|C|^{-1/2} \exp\!\left\{ -\frac12 \sum_{m=1}^{M} (\mathbf{x}_m-\boldsymbol{\mu})^T C^{-1} (\mathbf{x}_m-\boldsymbol{\mu}) \right\}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How plausible is observation x under mean μ and covariance Σ?

Key idea

Calculate a covariance-scaled distance from the mean and a normalization constant to obtain density.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the difference from the mean

    r=x−μr=x-\mu
  2. 2

    Adjust distance using variance or covariance

    d2=rTΣ−1rd^2=r^T\Sigma^{-1}r
  3. 3

    Calculate exponential decay

    exp⁡(−d2/2)\exp(-d^2/2)
  4. 4

    Divide by the normalization constant

    (2π)N/2∣Σ∣1/2(2\pi)^{N/2}|\Sigma|^{1/2}

Worked example

Work through the numbers — x=1 under a standard normal

A one-dimensional Gaussian with μ=0 and σ=1.

  1. 1

    Standardized distance

    z=(1−0)/1=1z=(1-0)/1=1
  2. 2

    Exponential term

    e−12/2=e−0.5≈0.6065e^{-1^2/2}=e^{-0.5}\approx0.6065
  3. 3

    Normalization

    0.6065/2π≈0.24200.6065/\sqrt{2\pi}\approx0.2420

Result: Density at x=1 is approximately 0.242. A density value is not itself an event probability.

Sanity check

Checks
  • Σ must be symmetric positive definite for the inverse and positive density to be stably defined.
  • Obtain probabilities through interval integration, distinct from pointwise density.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
X\mathbf{X}Data matrix collecting samples
μ\muAverage
∑\sumSum over all specified terms
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-023 Identifier: MAI7-023

The equation above preserves the source notation. Since each density contributes one |C|^{-1/2}, normalization for M independent vectors is corrected as follows.

Part I · Study equation

Gaussian density uses standardized distance from the mean
pX(x)=1(2π)MN/2∣C∣−M/2exp⁡ ⁣{−12∑m=1M(xm−μ)TC−1(xm−μ)}.p_{\mathbf{X}}(\mathbf{x}) = \frac{1}{(2\pi)^{MN/2}}|C|^{-M/2} \exp\!\left\{ -\frac12 \sum_{m=1}^{M} (\mathbf{x}_m-\boldsymbol{\mu})^T C^{-1} (\mathbf{x}_m-\boldsymbol{\mu}) \right\}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How plausible is observation x under mean μ and covariance Σ?

Key idea

Calculate a covariance-scaled distance from the mean and a normalization constant to obtain density.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the difference from the mean

    r=x−μr=x-\mu
  2. 2

    Adjust distance using variance or covariance

    d2=rTΣ−1rd^2=r^T\Sigma^{-1}r
  3. 3

    Calculate exponential decay

    exp⁡(−d2/2)\exp(-d^2/2)
  4. 4

    Divide by the normalization constant

    (2π)N/2∣Σ∣1/2(2\pi)^{N/2}|\Sigma|^{1/2}

Worked example

Work through the numbers — x=1 under a standard normal

A one-dimensional Gaussian with μ=0 and σ=1.

  1. 1

    Standardized distance

    z=(1−0)/1=1z=(1-0)/1=1
  2. 2

    Exponential term

    e−12/2=e−0.5≈0.6065e^{-1^2/2}=e^{-0.5}\approx0.6065
  3. 3

    Normalization

    0.6065/2π≈0.24200.6065/\sqrt{2\pi}\approx0.2420

Result: Density at x=1 is approximately 0.242. A density value is not itself an event probability.

Sanity check

Checks
  • Σ must be symmetric positive definite for the inverse and positive density to be stably defined.
  • Obtain probabilities through interval integration, distinct from pointwise density.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
X\mathbf{X}Data matrix collecting samples
μ\muAverage
∑\sumSum over all specified terms
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-023C Identifier: MAI7-023C

The lecture defines its two sample statistics as sums. Notice that the first lacks the 1/M of a conventional sample mean. Here, it is best read as a sum used to derive sufficient statistics.

Part I · Study equation

Probability calculations proceed through event definitions, conditions, and sum/product/normalization rules
μ^=∑m=1MXm.\widehat{\boldsymbol{\mu}} = \sum_{m=1}^{M}\mathbf{X}_m.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which event or observation interval's probability is required?

Key idea

Define the event, add mutually exclusive cases, and multiply chained conditions using conditional probabilities.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Translate events and conditions into words

    P(A|B) is the probability of A given B.

  2. 2

    Choose a sum rule or product rule

    Add mutually exclusive alternatives; multiply sequential conditions.

  3. 3

    Use total probability for the denominator if needed

  4. 4

    Check 0≤P≤1 and total probability 1

Worked example

Work through the numbers — Conditional probability

P(A∩B)=0.15 and P(B)=0.30.

  1. 1

    Apply the definition

    P(A∣B)=P(A∩B)P(B)P(A\mid B)=\frac{P(A\cap B)}{P(B)}
  2. 2

    Substitute values

    0.15/0.30=0.50.15/0.30=0.5

Result: Among cases where B occurs, A occurs half the time.

Sanity check

Checks
  • The conditioning event's probability in the denominator must be nonzero.
  • Distinguish density p(x) from event probability P(X∈A).

Symbols

Symbols used in this equation
SymbolMeaning
X\mathbf{X}Data matrix collecting samples
μ\muAverage
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-024 Identifier: MAI7-024

Part I · Study equation

Probability calculations proceed through event definitions, conditions, and sum/product/normalization rules
C^=∑m=1MXmXmT=XXT.\widehat{C} = \sum_{m=1}^{M}\mathbf{X}_m\mathbf{X}_m^T = \mathbf{X}\mathbf{X}^T.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which event or observation interval's probability is required?

Key idea

Define the event, add mutually exclusive cases, and multiply chained conditions using conditional probabilities.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Translate events and conditions into words

    P(A|B) is the probability of A given B.

  2. 2

    Choose a sum rule or product rule

    Add mutually exclusive alternatives; multiply sequential conditions.

  3. 3

    Use total probability for the denominator if needed

  4. 4

    Check 0≤P≤1 and total probability 1

Worked example

Work through the numbers — Conditional probability

P(A∩B)=0.15 and P(B)=0.30.

  1. 1

    Apply the definition

    P(A∣B)=P(A∩B)P(B)P(A\mid B)=\frac{P(A\cap B)}{P(B)}
  2. 2

    Substitute values

    0.15/0.30=0.50.15/0.30=0.5

Result: Among cases where B occurs, A occurs half the time.

Sanity check

Checks
  • The conditioning event's probability in the denominator must be nonzero.
  • Distinguish density p(x) from event probability P(X∈A).

Symbols

Symbols used in this equation
SymbolMeaning
X\mathbf{X}Data matrix collecting samples
∑\sumSum over all specified terms
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-025 Identifier: MAI7-025

Part I · Study equation

Gaussian density uses standardized distance from the mean
pX(x)=1(2π)MN/2∣C∣−1/2exp⁡ ⁣{−12tr⁡(C^C−1)+μ^TC−1μ−M2μTC−1μ}.p_{\mathbf{X}}(\mathbf{x}) = \frac{1}{(2\pi)^{MN/2}}|C|^{-1/2} \exp\!\left\{ -\frac12\operatorname{tr}(\widehat{C}C^{-1}) + \widehat{\boldsymbol{\mu}}^TC^{-1}\boldsymbol{\mu} - \frac{M}{2}\boldsymbol{\mu}^TC^{-1}\boldsymbol{\mu} \right\}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How plausible is observation x under mean μ and covariance Σ?

Key idea

Calculate a covariance-scaled distance from the mean and a normalization constant to obtain density.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the difference from the mean

    r=x−μr=x-\mu
  2. 2

    Adjust distance using variance or covariance

    d2=rTΣ−1rd^2=r^T\Sigma^{-1}r
  3. 3

    Calculate exponential decay

    exp⁡(−d2/2)\exp(-d^2/2)
  4. 4

    Divide by the normalization constant

    (2π)N/2∣Σ∣1/2(2\pi)^{N/2}|\Sigma|^{1/2}

Worked example

Work through the numbers — x=1 under a standard normal

A one-dimensional Gaussian with μ=0 and σ=1.

  1. 1

    Standardized distance

    z=(1−0)/1=1z=(1-0)/1=1
  2. 2

    Exponential term

    e−12/2=e−0.5≈0.6065e^{-1^2/2}=e^{-0.5}\approx0.6065
  3. 3

    Normalization

    0.6065/2π≈0.24200.6065/\sqrt{2\pi}\approx0.2420

Result: Density at x=1 is approximately 0.242. A density value is not itself an event probability.

Sanity check

Checks
  • Σ must be symmetric positive definite for the inverse and positive density to be stably defined.
  • Obtain probabilities through interval integration, distinct from pointwise density.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
X\mathbf{X}Data matrix collecting samples
μ\muAverage
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-026 Identifier: MAI7-026

Part I · Study equation

Identify the equation's role and symbols
tr⁡(AB)=tr⁡(BA).\operatorname{tr}(AB)=\operatorname{tr}(BA).
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-027 Identifier: MAI7-027

Part I · Study equation

Probability calculations proceed through event definitions, conditions, and sum/product/normalization rules
tr⁡(A)=∑iAi,i.\operatorname{tr}(A)=\sum_i A_{i,i}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which event or observation interval's probability is required?

Key idea

Define the event, add mutually exclusive cases, and multiply chained conditions using conditional probabilities.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Translate events and conditions into words

    P(A|B) is the probability of A given B.

  2. 2

    Choose a sum rule or product rule

    Add mutually exclusive alternatives; multiply sequential conditions.

  3. 3

    Use total probability for the denominator if needed

  4. 4

    Check 0≤P≤1 and total probability 1

Worked example

Work through the numbers — Conditional probability

P(A∩B)=0.15 and P(B)=0.30.

  1. 1

    Apply the definition

    P(A∣B)=P(A∩B)P(B)P(A\mid B)=\frac{P(A\cap B)}{P(B)}
  2. 2

    Substitute values

    0.15/0.30=0.50.15/0.30=0.5

Result: Among cases where B occurs, A occurs half the time.

Sanity check

Checks
  • The conditioning event's probability in the denominator must be nonzero.
  • Distinguish density p(x) from event probability P(X∈A).

Symbols

Symbols used in this equation
SymbolMeaning
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-028 Identifier: MAI7-028

#7. Two-dimensional discrete random processes

Part I · Study equation

Identify the equation's role and symbols
Xs,s=(s1,s2)∈Z2,S⊆Z2.X_{\mathbf{s}}, \qquad \mathbf{s}=(s_1,s_2)\in\mathbb{Z}^2, \qquad \mathcal{S}\subseteq\mathbb{Z}^2.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-029 Identifier: MAI7-029

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
μs=E[Xs].\mu_{\mathbf{s}}=E[X_{\mathbf{s}}].
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
μ\muAverage
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-030 Identifier: MAI7-030

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
Rs,r=E[XsXr∗].R_{\mathbf{s},\mathbf{r}} = E[X_{\mathbf{s}}X_{\mathbf{r}}^*].
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-031 Identifier: MAI7-031

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
Cs,r=E[(Xs−μs)(Xr−μr)∗]=Rs,r−μsμr∗.C_{\mathbf{s},\mathbf{r}} = E[(X_{\mathbf{s}}-\mu_{\mathbf{s}}) (X_{\mathbf{r}}-\mu_{\mathbf{r}})^*] = R_{\mathbf{s},\mathbf{r}} - \mu_{\mathbf{s}}\mu_{\mathbf{r}}^*.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
μ\muAverage
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-032 Identifier: MAI7-032

#8. WSS

For a second-order random process, shift-invariant mean and covariance define wide-sense stationarity.

Part I · Study equation

Identify the equation's role and symbols
μs=μ(0,0).\mu_{\mathbf{s}}=\mu_{(0,0)}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
μ\muAverage

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-033 Identifier: MAI7-033

Part I · Study equation

Identify the equation's role and symbols
Cr,r+s=C(0,0),s.C_{\mathbf{r},\mathbf{r}+\mathbf{s}} = C_{(0,0),\mathbf{s}}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-034 Identifier: MAI7-034

The WSS properties are listed in full.

Part I · Study equation

Expectation is the long-run average formed by summing values times probabilities
Rs,(0,0)=E[XsX(0,0)∗]=E[Xs+rXr∗],∀r∈Z2.R_{\mathbf{s},(0,0)} = E[X_{\mathbf{s}}X_{(0,0)}^*] = E[X_{\mathbf{s}+\mathbf{r}}X_{\mathbf{r}}^*], \qquad\forall\mathbf{r}\in\mathbb{Z}^2.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Where does the average tend after many repetitions of the same experiment?

Key idea

Multiply each outcome by its probability and sum. For continuous variables, integrate instead.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Pair possible values with their probabilities

    (xi,pi)(x_i,p_i)
  2. 2

    Multiply each value by its probability

    xipix_ip_i
  3. 3

    Add all terms

    E[X]=∑ixipiE[X]=\sum_i x_ip_i
  4. 4

    Check that probabilities sum to one

    ∑ipi=1\sum_i p_i=1

Worked example

Work through the numbers — Mean of a discrete random variable

X takes 1,2,3 with probabilities 0.2,0.5,0.3 respectively.

  1. 1

    Weighted values

    1(0.2), 2(0.5), 3(0.3)1(0.2),\ 2(0.5),\ 3(0.3)
  2. 2

    Sum

    0.2+1.0+0.9=2.10.2+1.0+0.9=2.1

Result: E[X]=2.1. It need not be an observable outcome itself.

Sanity check

Checks
  • Probabilities or densities must be normalized.
  • E[g(X)] generally differs from g(E[X]).

Symbols

Symbols used in this equation
SymbolMeaning
E[⋅]E[\cdot]Expectation: the probability-weighted average of possible values

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-035 Identifier: MAI7-035

Part I · Study equation

Identify the equation's role and symbols
Ys=Xs+s0.Y_{\mathbf{s}}=X_{\mathbf{s}+\mathbf{s}_0}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-036 Identifier: MAI7-036

Part I · Study equation

Identify the equation's role and symbols
μY=μX,RY=RX.\mu_Y=\mu_X,\qquad R_Y=R_X.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Symbols

Symbols used in this equation
SymbolMeaning
μ\muAverage

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-037 Identifier: MAI7-037

Part I · Study equation

Variance averages squared deviations from the mean
Cs,(0,0)=Var⁡(Xs).C_{\mathbf{s},(0,0)} = \operatorname{Var}(X_{\mathbf{s}}).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How widely does a random variable spread around its mean?

Key idea

Subtract the mean from each value, square, and take the probability-weighted average.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the mean

    μ=E[X]\mu=E[X]
  2. 2

    Calculate deviations from the mean

    X−μX-\mu
  3. 3

    Square the deviations

    (X−μ)2(X-\mu)^2
  4. 4

    Take the expectation

    Var⁡(X)=E[(X−μ)2]\operatorname{Var}(X)=E[(X-\mu)^2]

Worked example

Work through the numbers — X∈{1,2,3} with probabilities {0.2,0.5,0.3}

Use mean μ=2.1 from the preceding example.

  1. 1

    Squared deviations

    (1−2.1)2=1.21, (2−2.1)2=0.01, (3−2.1)2=0.81(1-2.1)^2=1.21,\ (2-2.1)^2=0.01,\ (3-2.1)^2=0.81
  2. 2

    Probability-weighted sum

    0.2(1.21)+0.5(0.01)+0.3(0.81)=0.490.2(1.21)+0.5(0.01)+0.3(0.81)=0.49

Result: Variance is 0.49 and standard deviation is √0.49=0.7.

Sanity check

Checks
  • Variance cannot be negative.
  • Standard deviation shares the variable's units; variance has squared units.

Symbols

Symbols used in this equation
SymbolMeaning
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-038 Identifier: MAI7-038

Part I · Study equation

Variance averages squared deviations from the mean
Rs,(0,0)=Var⁡(Xs)+∣μs∣2.R_{\mathbf{s},(0,0)} = \operatorname{Var}(X_{\mathbf{s}}) + |\mu_{\mathbf{s}}|^2.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How widely does a random variable spread around its mean?

Key idea

Subtract the mean from each value, square, and take the probability-weighted average.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the mean

    μ=E[X]\mu=E[X]
  2. 2

    Calculate deviations from the mean

    X−μX-\mu
  3. 3

    Square the deviations

    (X−μ)2(X-\mu)^2
  4. 4

    Take the expectation

    Var⁡(X)=E[(X−μ)2]\operatorname{Var}(X)=E[(X-\mu)^2]

Worked example

Work through the numbers — X∈{1,2,3} with probabilities {0.2,0.5,0.3}

Use mean μ=2.1 from the preceding example.

  1. 1

    Squared deviations

    (1−2.1)2=1.21, (2−2.1)2=0.01, (3−2.1)2=0.81(1-2.1)^2=1.21,\ (2-2.1)^2=0.01,\ (3-2.1)^2=0.81
  2. 2

    Probability-weighted sum

    0.2(1.21)+0.5(0.01)+0.3(0.81)=0.490.2(1.21)+0.5(0.01)+0.3(0.81)=0.49

Result: Variance is 0.49 and standard deviation is √0.49=0.7.

Sanity check

Checks
  • Variance cannot be negative.
  • Standard deviation shares the variable's units; variance has squared units.

Symbols

Symbols used in this equation
SymbolMeaning
μ\muAverage

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-039 Identifier: MAI7-039

Part I · Study equation

Identify the equation's role and symbols
R−s,(0,0)=Rs,(0,0)∗.R_{-\mathbf{s},(0,0)} = R_{\mathbf{s},(0,0)}^*.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-040 Identifier: MAI7-040

Part I · Study equation

Identify the equation's role and symbols
R−s,(0,0)=R(0,0)−s,(0,0)=R(0,0),s=Rs,(0,0)∗.R_{-\mathbf{s},(0,0)} = R_{(0,0)-\mathbf{s},(0,0)} = R_{(0,0),\mathbf{s}} = R_{\mathbf{s},(0,0)}^*.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-041 Identifier: MAI7-041

Part I · Study equation

Identify the equation's role and symbols
∣Rs∣≤Rs,(0,0).|R_{\mathbf{s}}|\le R_{\mathbf{s},(0,0)}.
Explanation Core explanation

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI7-042 Identifier: MAI7-042

#9. Two-dimensional power spectral density

The power spectrum is the DSFT of autocorrelation, rather than a directly defined DSFT of the WSS process itself.

Part I · Study equation

Probability calculations proceed through event definitions, conditions, and sum/product/normalization rules
SX(eiμ,eiν)=∑m=−∞∞∑n=−∞∞R(m,n),(0,0)e−i(mμ+nν).S_X(e^{i\mu},e^{i\nu}) = \sum_{m=-\infty}^{\infty} \sum_{n=-\infty}^{\infty} R_{(m,n),(0,0)} e^{-i(m\mu+n\nu)}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

Which event or observation interval's probability is required?

Key idea

Define the event, add mutually exclusive cases, and multiply chained conditions using conditional probabilities.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Translate events and conditions into words

    P(A|B) is the probability of A given B.

  2. 2

    Choose a sum rule or product rule

    Add mutually exclusive alternatives; multiply sequential conditions.

  3. 3

    Use total probability for the denominator if needed

  4. 4

    Check 0≤P≤1 and total probability 1

Worked example

Work through the numbers — Conditional probability

P(A∩B)=0.15 and P(B)=0.30.

  1. 1

    Apply the definition

    P(A∣B)=P(A∩B)P(B)P(A\mid B)=\frac{P(A\cap B)}{P(B)}
  2. 2

    Substitute values

    0.15/0.30=0.50.15/0.30=0.5

Result: Among cases where B occurs, A occurs half the time.

Sanity check

Checks
  • The conditioning event's probability in the denominator must be nonzero.
  • Distinguish density p(x) from event probability P(X∈A).

Symbols

Symbols used in this equation
SymbolMeaning
μ\muAverage
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-043 Identifier: MAI7-043

This relationship is the core of the Wiener–Khinchin theorem.

#9.1 A zero-mean WSS white process

Part I · Study equation

Variance averages squared deviations from the mean
R(m,n),(0,0)=C(m,n),(0,0)=σ2δ[m,n].R_{(m,n),(0,0)} = C_{(m,n),(0,0)} = \sigma^2\delta[m,n].
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How widely does a random variable spread around its mean?

Key idea

Subtract the mean from each value, square, and take the probability-weighted average.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the mean

    μ=E[X]\mu=E[X]
  2. 2

    Calculate deviations from the mean

    X−μX-\mu
  3. 3

    Square the deviations

    (X−μ)2(X-\mu)^2
  4. 4

    Take the expectation

    Var⁡(X)=E[(X−μ)2]\operatorname{Var}(X)=E[(X-\mu)^2]

Worked example

Work through the numbers — X∈{1,2,3} with probabilities {0.2,0.5,0.3}

Use mean μ=2.1 from the preceding example.

  1. 1

    Squared deviations

    (1−2.1)2=1.21, (2−2.1)2=0.01, (3−2.1)2=0.81(1-2.1)^2=1.21,\ (2-2.1)^2=0.01,\ (3-2.1)^2=0.81
  2. 2

    Probability-weighted sum

    0.2(1.21)+0.5(0.01)+0.3(0.81)=0.490.2(1.21)+0.5(0.01)+0.3(0.81)=0.49

Result: Variance is 0.49 and standard deviation is √0.49=0.7.

Sanity check

Checks
  • Variance cannot be negative.
  • Standard deviation shares the variable's units; variance has squared units.

Symbols

Symbols used in this equation
SymbolMeaning
σ\sigmaStandard deviation or sigmoid notation
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-044 Identifier: MAI7-044

Part I · Study equation

Variance averages squared deviations from the mean
SX(eiμ,eiν)=σ2.S_X(e^{i\mu},e^{i\nu})=\sigma^2.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How widely does a random variable spread around its mean?

Key idea

Subtract the mean from each value, square, and take the probability-weighted average.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the mean

    μ=E[X]\mu=E[X]
  2. 2

    Calculate deviations from the mean

    X−μX-\mu
  3. 3

    Square the deviations

    (X−μ)2(X-\mu)^2
  4. 4

    Take the expectation

    Var⁡(X)=E[(X−μ)2]\operatorname{Var}(X)=E[(X-\mu)^2]

Worked example

Work through the numbers — X∈{1,2,3} with probabilities {0.2,0.5,0.3}

Use mean μ=2.1 from the preceding example.

  1. 1

    Squared deviations

    (1−2.1)2=1.21, (2−2.1)2=0.01, (3−2.1)2=0.81(1-2.1)^2=1.21,\ (2-2.1)^2=0.01,\ (3-2.1)^2=0.81
  2. 2

    Probability-weighted sum

    0.2(1.21)+0.5(0.01)+0.3(0.81)=0.490.2(1.21)+0.5(0.01)+0.3(0.81)=0.49

Result: Variance is 0.49 and standard deviation is √0.49=0.7.

Sanity check

Checks
  • Variance cannot be negative.
  • Standard deviation shares the variable's units; variance has squared units.

Symbols

Symbols used in this equation
SymbolMeaning
μ\muAverage
σ\sigmaStandard deviation or sigmoid notation

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-045 Identifier: MAI7-045

#9.2 Nonzero mean

Part I · Study equation

Variance averages squared deviations from the mean
C(m,n),(0,0)=σ2δ[m,n].C_{(m,n),(0,0)}=\sigma^2\delta[m,n].
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How widely does a random variable spread around its mean?

Key idea

Subtract the mean from each value, square, and take the probability-weighted average.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the mean

    μ=E[X]\mu=E[X]
  2. 2

    Calculate deviations from the mean

    X−μX-\mu
  3. 3

    Square the deviations

    (X−μ)2(X-\mu)^2
  4. 4

    Take the expectation

    Var⁡(X)=E[(X−μ)2]\operatorname{Var}(X)=E[(X-\mu)^2]

Worked example

Work through the numbers — X∈{1,2,3} with probabilities {0.2,0.5,0.3}

Use mean μ=2.1 from the preceding example.

  1. 1

    Squared deviations

    (1−2.1)2=1.21, (2−2.1)2=0.01, (3−2.1)2=0.81(1-2.1)^2=1.21,\ (2-2.1)^2=0.01,\ (3-2.1)^2=0.81
  2. 2

    Probability-weighted sum

    0.2(1.21)+0.5(0.01)+0.3(0.81)=0.490.2(1.21)+0.5(0.01)+0.3(0.81)=0.49

Result: Variance is 0.49 and standard deviation is √0.49=0.7.

Sanity check

Checks
  • Variance cannot be negative.
  • Standard deviation shares the variable's units; variance has squared units.

Symbols

Symbols used in this equation
SymbolMeaning
σ\sigmaStandard deviation or sigmoid notation
CCTotal class count or a matrix; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-046 Identifier: MAI7-046

Part I · Study equation

Variance averages squared deviations from the mean
R(m,n),(0,0)=σ2δ[m,n]+∣μ(0,0)∣2.R_{(m,n),(0,0)} = \sigma^2\delta[m,n] + |\mu_{(0,0)}|^2.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How widely does a random variable spread around its mean?

Key idea

Subtract the mean from each value, square, and take the probability-weighted average.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the mean

    μ=E[X]\mu=E[X]
  2. 2

    Calculate deviations from the mean

    X−μX-\mu
  3. 3

    Square the deviations

    (X−μ)2(X-\mu)^2
  4. 4

    Take the expectation

    Var⁡(X)=E[(X−μ)2]\operatorname{Var}(X)=E[(X-\mu)^2]

Worked example

Work through the numbers — X∈{1,2,3} with probabilities {0.2,0.5,0.3}

Use mean μ=2.1 from the preceding example.

  1. 1

    Squared deviations

    (1−2.1)2=1.21, (2−2.1)2=0.01, (3−2.1)2=0.81(1-2.1)^2=1.21,\ (2-2.1)^2=0.01,\ (3-2.1)^2=0.81
  2. 2

    Probability-weighted sum

    0.2(1.21)+0.5(0.01)+0.3(0.81)=0.490.2(1.21)+0.5(0.01)+0.3(0.81)=0.49

Result: Variance is 0.49 and standard deviation is √0.49=0.7.

Sanity check

Checks
  • Variance cannot be negative.
  • Standard deviation shares the variable's units; variance has squared units.

Symbols

Symbols used in this equation
SymbolMeaning
μ\muAverage
σ\sigmaStandard deviation or sigmoid notation

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-047 Identifier: MAI7-047

Part I · Study equation

Gaussian density uses standardized distance from the mean
SX(eiμ,eiν)=σ2+∣μ(0,0)∣2(2π)2∑k,l=−∞∞δ(μ−2kπ,ν−2lπ).S_X(e^{i\mu},e^{i\nu}) = \sigma^2 + |\mu_{(0,0)}|^2(2\pi)^2 \sum_{k,l=-\infty}^{\infty} \delta(\mu-2k\pi,\nu-2l\pi).
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How plausible is observation x under mean μ and covariance Σ?

Key idea

Calculate a covariance-scaled distance from the mean and a normalization constant to obtain density.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the difference from the mean

    r=x−μr=x-\mu
  2. 2

    Adjust distance using variance or covariance

    d2=rTΣ−1rd^2=r^T\Sigma^{-1}r
  3. 3

    Calculate exponential decay

    exp⁡(−d2/2)\exp(-d^2/2)
  4. 4

    Divide by the normalization constant

    (2π)N/2∣Σ∣1/2(2\pi)^{N/2}|\Sigma|^{1/2}

Worked example

Work through the numbers — x=1 under a standard normal

A one-dimensional Gaussian with μ=0 and σ=1.

  1. 1

    Standardized distance

    z=(1−0)/1=1z=(1-0)/1=1
  2. 2

    Exponential term

    e−12/2=e−0.5≈0.6065e^{-1^2/2}=e^{-0.5}\approx0.6065
  3. 3

    Normalization

    0.6065/2π≈0.24200.6065/\sqrt{2\pi}\approx0.2420

Result: Density at x=1 is approximately 0.242. A density value is not itself an event probability.

Sanity check

Checks
  • Σ must be symmetric positive definite for the inverse and positive density to be stably defined.
  • Obtain probabilities through interval integration, distinct from pointwise density.

Symbols

Symbols used in this equation
SymbolMeaning
μ\muAverage
σ\sigmaStandard deviation or sigmoid notation
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Probability ↗ Check formal definitions of random variables, expectation, variance, and Bayes' rule. Seeing Theory ↗ Explore probability, distributions, and Bayesian inference with interactive illustrations.
Modern Artificial Intelligence I · MAI7-048 Identifier: MAI7-048

#10. Implementation perspective in 2026

Frequency analysis remains a direct tool for two-dimensional images. As of the source’s August 2026 update, PyTorch 2.13’s torch.fft.fft2 provides the 2D DFT, and its documentation notes Hermitian symmetry in Fourier representations of real signals.

WSS does not assert that a real finite image is a perfectly stationary finite object. As the lecture notes, it models a finite image as part of an observed infinite WSS process.

#Part 1.7: Losses, Gradient Descent, Minibatches, and Generalization

#1. Minimize expected loss

In statistical machine learning, model D maps input x to prediction D(x), and a loss measures its discrepancy from target y.

Versions of the lecture number this equation (4) or (5). This series tracks the equation itself, rather than relying on numbering alone.

Part I · Study equation

Identify the equation's role and symbols
argmin⁡D  Ex,yL(D(x),y).\underset{D}{\operatorname{argmin}}\; E_{\mathbf{x},\mathbf{y}} \mathcal{L}(D(\mathbf{x}),\mathbf{y}).
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

gradient field·step size·mini-batch update

Run gradient descent on a 2D positive-definite quadratic, inspect the stable step-size bound, and calculate minibatch sums of sample gradients in the same view.

Structure explanation
Loading the lesson matched to this equation and derivation step.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI8-001 Identifier: MAI8-001

#2. Approximate expectations with finite-sample averages

The true joint data distribution is usually unknown. Training samples (\mathbf{x}_l,\mathbf{y}_l) approximate expected loss through an empirical average.

Part I · Study equation

Identify the equation's role and symbols
Ex,yL(D(x),y)≈1L∑l=1LL(D(xl),yl).E_{\mathbf{x},\mathbf{y}} \mathcal{L}(D(\mathbf{x}),\mathbf{y}) \approx \frac1L\sum_{l=1}^{L} \mathcal{L}(D(\mathbf{x}_l),\mathbf{y}_l).
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

gradient field·step size·mini-batch update

Run gradient descent on a 2D positive-definite quadratic, inspect the stable step-size bound, and calculate minibatch sums of sample gradients in the same view.

Synchronized derivation
Loading the lesson matched to this equation and derivation step.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector
∑\sumSum over all specified terms
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI8-002 Identifier: MAI8-002

The March 19, 2025 version presents the right side as ?? for students to complete. The expression above is completed using the earlier version and surrounding context.

#3. MSE example

Part I · Study equation

MSE has three steps: residuals, squares, and averaging
argmin⁡D  Ex,yL(D(x),y),L(D(x),y)=∥D(x)−y∥22.\underset{D}{\operatorname{argmin}}\; E_{\mathbf{x},\mathbf{y}} \mathcal{L}(D(\mathbf{x}),\mathbf{y}), \qquad \mathcal{L}(D(\mathbf{x}),\mathbf{y}) = \|D(\mathbf{x})-\mathbf{y}\|_2^2.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How far are predictions from targets on average?

Key idea

Subtract targets from predictions, square each residual, and divide by sample count. Larger errors receive greater penalties.

Formula-specific lesson

gradient field·step size·mini-batch update

Run gradient descent on a 2D positive-definite quadratic, inspect the stable step-size bound, and calculate minibatch sums of sample gradients in the same view.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate predictions

    For linear regression, predictions are Xw or wᵀx.

    y^=Xw\hat y=Xw
  2. 2

    Calculate residuals

    r=y^−yr=\hat y-y
  3. 3

    Square the residuals

    Remove signs and emphasize larger errors.

    ri2r_i^2
  4. 4

    Sum and divide by sample count

    MSE=1L∑i=1Lri2\mathrm{MSE}=\frac{1}{L}\sum_{i=1}^{L}r_i^2

Worked example

Work through the numbers — MSE for two samples

Compare predictions and targets directly.

  1. 1

    Targets and predictions

    y=[3,1],y^=[2,4]y=[3,1],\quad\hat y=[2,4]
  2. 2

    Residual

    y^−y=[−1,3]\hat y-y=[-1,3]
  3. 3

    Square

    [−1,3]2=[1,9][-1,3]^2=[1,9]
  4. 4

    Average

    (1+9)/2=5(1+9)/2=5

Result: MSE is 5. The second sample's error of 3 substantially increases it.

Sanity check

Checks
  • Check whether the expression divides by sample count or is merely a sum of squares.
  • MSE has squared output units.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI8-003 Identifier: MAI8-003

Part I · Study equation

A norm summarizes many components as one magnitude
∥x∥22=∑n∣xn∣2.\|\mathbf{x}\|_2^2 = \sum_n |x_n|^2.
Explanation and calculations 4 calculation steps · Numerical example · Checks included

The question this equation answers

How far is a vector or error from the origin?

Key idea

The L2 norm sums squared components and takes the square root. Squared norm omits the final square root.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Square each component

    Turn both negative and positive components into nonnegative contributions.

  2. 2

    Sum the squares

    ∑ixi2\sum_i x_i^2
  3. 3

    Take the square root for a norm

    ∥x∥2=∑ixi2\lVert x\rVert_2=\sqrt{\sum_i x_i^2}
  4. 4

    Distinguish the squared norm

    ∥x∥22=∑ixi2\lVert x\rVert_2^2=\sum_i x_i^2

Worked example

Work through the numbers — The 3–4–5 vector

A familiar Pythagorean example.

  1. 1

    Vector

    x=[3,4]Tx=[3,4]^T
  2. 2

    Sum of squares

    32+42=9+16=253^2+4^2=9+16=25
  3. 3

    Square root

    25=5\sqrt{25}=5

Result: Thus ||x||₂=5 and ||x||₂²=25.

Sanity check

Checks
  • A norm is always nonnegative.
  • Confusing a norm with its square changes units and derivative coefficients.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∑\sumSum over all specified terms

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas.
Modern Artificial Intelligence I · MAI8-004 Identifier: MAI8-004

#4. Gradient

The gradient of f:\mathbb{R}^N\to\mathbb{R} points toward steepest increase and is perpendicular to level curves.

Part I · Study equation

A gradient describes the direction and rate of loss change
∇f(z)=[∂f∂x1(z)⋯∂f∂xN(z)]T.\nabla f(\mathbf{z}) = \begin{bmatrix} \frac{\partial f}{\partial x_1}(\mathbf{z})& \cdots& \frac{\partial f}{\partial x_N}(\mathbf{z}) \end{bmatrix}^{T}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Formula-specific lesson

gradient field·step size·mini-batch update

Run gradient descent on a 2D positive-definite quadratic, inspect the stable step-size bound, and calculate minibatch sums of sample gradients in the same view.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
∇\nablaA gradient collecting derivatives in each variable direction
∂\partialPartial derivative with respect to a particular variable

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI8-005 Identifier: MAI8-005

#5. Conditions for a local minimum

Part I · Study equation

A gradient describes the direction and rate of loss change
∇f(x∗)=0.\nabla f(\mathbf{x}^*)=0.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Formula-specific lesson

gradient field·step size·mini-batch update

Run gradient descent on a 2D positive-definite quadratic, inspect the stable step-size bound, and calculate minibatch sums of sample gradients in the same view.

Synchronized derivation
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∇\nablaA gradient collecting derivatives in each variable direction

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI8-006 Identifier: MAI8-006

Part I · Study equation

A gradient describes the direction and rate of loss change
yT∇2f(x∗)y>0,∀y≠0, y∈RN.\mathbf{y}^T\nabla^2f(\mathbf{x}^*)\mathbf{y}>0, \qquad \forall\mathbf{y}\neq0,\ \mathbf{y}\in\mathbb{R}^{N}.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Formula-specific lesson

gradient field·step size·mini-batch update

Run gradient descent on a 2D positive-definite quadratic, inspect the stable step-size bound, and calculate minibatch sums of sample gradients in the same view.

Structure explanation
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector
∇\nablaA gradient collecting derivatives in each variable direction

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI8-007 Identifier: MAI8-007

Part I · Study equation

A gradient describes the direction and rate of loss change
[∇2f(x)]m,n=∂2f∂xm∂xn(x).\left[\nabla^2f(\mathbf{x})\right]_{m,n} = \frac{\partial^2f}{\partial x_m\partial x_n}(\mathbf{x}).
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Formula-specific lesson

gradient field·step size·mini-batch update

Run gradient descent on a 2D positive-definite quadratic, inspect the stable step-size bound, and calculate minibatch sums of sample gradients in the same view.

Structure explanation
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∇\nablaA gradient collecting derivatives in each variable direction
∂\partialPartial derivative with respect to a particular variable

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI8-008 Identifier: MAI8-008

#6. Gradient descent

Part I · Study equation

Gradient descent steps opposite the gradient from the current point
x(i+1)=x(i)−α∇f(x(i)).\mathbf{x}^{(i+1)} = \mathbf{x}^{(i)} - \alpha\nabla f(\mathbf{x}^{(i)}).
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

Where should parameters move next to reduce loss?

Key idea

Calculate the gradient at current parameters, multiply by the learning rate, and subtract to obtain new parameters.

Formula-specific lesson

gradient field·step size·mini-batch update

Run gradient descent on a 2D positive-definite quadratic, inspect the stable step-size bound, and calculate minibatch sums of sample gradients in the same view.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Calculate the current loss gradient

    gk=∇J(wk)g_k=\nabla J(w_k)
  2. 2

    Determine the step size

    Δw=αgk\Delta w=\alpha g_k
  3. 3

    Move opposite the direction of increase

    wk+1=wk−αgkw_{k+1}=w_k-\alpha g_k
  4. 4

    Check the new loss and repeat

    Too large a learning rate can diverge; too small is very slow.

Worked example

Work through the numbers — One update

f(w)=(w-3)², current w=5, and learning rate α=0.1.

  1. 1

    Gradient

    g=2(5−3)=4g=2(5-3)=4
  2. 2

    Step size

    αg=0.1⋅4=0.4\alpha g=0.1\cdot4=0.4
  3. 3

    Update

    wnew=5−0.4=4.6w_{new}=5-0.4=4.6
  4. 4

    Compare losses

    f(5)=4,f(4.6)=2.56f(5)=4,\qquad f(4.6)=2.56

Result: One step reduces loss from 4 to 2.56.

Sanity check

Checks
  • Check for the minus sign.
  • Recording loss before and after updates alongside gradient norm helps detect divergence early.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
α\alphaLearning rate, mixture coefficient, or dual coefficient
∇\nablaA gradient collecting derivatives in each variable direction

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation. Deep Learning Book · Optimization ↗ Review the full connection between losses and gradient-based learning.
Modern Artificial Intelligence I · MAI8-009 Identifier: MAI8-009

A learning rate \alpha that is too small slows convergence; one that is too large can cause divergence.

Computing each gradient from the full dataset is expensive for large L. The MSE example requires the following gradient across every sample.

Part I · Study equation

A gradient describes the direction and rate of loss change
∇D∥D(xl)−yl∥22,l=1,…,L.\nabla_D \left\| D(\mathbf{x}_l)-\mathbf{y}_l \right\|_2^2, \qquad l=1,\ldots,L.
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Formula-specific lesson

gradient field·step size·mini-batch update

Run gradient descent on a 2D positive-definite quadratic, inspect the stable step-size bound, and calculate minibatch sums of sample gradients in the same view.

Synchronized derivation
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
y\mathbf{y}Target or observation vector
∇\nablaA gradient collecting derivatives in each variable direction
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI8-010 Identifier: MAI8-010

#7. Stochastic minibatch gradient descent

First shuffle sample indices randomly and divide them into batches of size B.

Part I · Study equation

Identify the equation's role and symbols
{1,…,L}=L(1)∪⋯∪L(L/B),∣L(i)∣=B.\{1,\ldots,L\} = \mathcal{L}^{(1)} \cup\cdots\cup \mathcal{L}^{(L/B)}, \qquad |\mathcal{L}^{(i)}|=B.
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

gradient field·step size·mini-batch update

Run gradient descent on a 2D positive-definite quadratic, inspect the stable step-size bound, and calculate minibatch sums of sample gradients in the same view.

Structure explanation
Loading the lesson matched to this equation and derivation step.

Symbols

Symbols used in this equation
SymbolMeaning
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI8-011 Identifier: MAI8-011

Update for each batch as follows.

Part I · Study equation

A gradient describes the direction and rate of loss change
x(i+1)←x(i)−α∑l∈L(i+1)∇fl(x(i)).\mathbf{x}^{(i+1)} \leftarrow \mathbf{x}^{(i)} - \alpha \sum_{l\in\mathcal{L}^{(i+1)}} \nabla f_l(\mathbf{x}^{(i)}).
Explanation and calculations 4 calculation steps · Numerical example · Checks included · Formula-specific lesson

The question this equation answers

How much does the objective increase or decrease when each parameter increases?

Key idea

Collect partial derivatives into a vector and move in the opposite direction during minimization.

Formula-specific lesson

gradient field·step size·mini-batch update

Run gradient descent on a 2D positive-definite quadratic, inspect the stable step-size bound, and calculate minibatch sums of sample gradients in the same view.

Interactive
Loading the lesson matched to this equation and derivation step.

Calculation walkthrough

Calculation steps
Intermediate calculations included
  1. 1

    Choose the differentiation variable

    Treat other variables as constants.

  2. 2

    Apply sum, product, and chain rules

    For composition, multiply the outer derivative by the inner derivative.

  3. 3

    Collect partial derivatives into one vector

    ∇f=[∂f/∂x1,…,∂f/∂xN]T\nabla f=[\partial f/\partial x_1,\ldots,\partial f/\partial x_N]^T
  4. 4

    Substitute current values

    Obtain numerical gradient magnitudes and signs.

Worked example

Work through the numbers — Gradient of a one-dimensional loss

The loss squares distance from target value 3.

  1. 1

    Function

    f(w)=(w−3)2f(w)=(w-3)^2
  2. 2

    Derivative

    f′(w)=2(w−3)f'(w)=2(w-3)
  3. 3

    Substitute w=5

    f′(5)=2(5−3)=4f'(5)=2(5-3)=4

Result: The positive gradient 4 means decreasing w reduces loss.

Sanity check

Checks
  • The gradient and parameter vectors must have matching lengths.
  • A constant term has derivative zero.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
α\alphaLearning rate, mixture coefficient, or dual coefficient
∇\nablaA gradient collecting derivatives in each variable direction
∑\sumSum over all specified terms
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book · Linear Algebra ↗ Review vectors, matrices, inner products, and inverses in a consistent notation. The Matrix Cookbook ↗ A useful quick reference for matrix identities and derivative formulas. Deep Learning Book · Numerical Computation ↗ Connect gradients, condition numbers, and numerical stability through computation.
Modern Artificial Intelligence I · MAI8-012 Identifier: MAI8-012

The lecture defines total empirical loss by this sum.

Part I · Study equation

Identify the equation's role and symbols
∑l=1Lfl(x).\sum_{l=1}^{L}f_l(\mathbf{x}).
Explanation Core explanation · Formula-specific lesson

The question this equation answers

What does this equation define or connect in the article?

Key idea

For equations without an actual calculation or derivation, briefly clarify context and symbols rather than attaching a generic procedure.

Formula-specific lesson

gradient field·step size·mini-batch update

Run gradient descent on a 2D positive-definite quadratic, inspect the stable step-size bound, and calculate minibatch sums of sample gradients in the same view.

Synchronized derivation
Loading the lesson matched to this equation and derivation step.

Symbols

Symbols used in this equation
SymbolMeaning
x\mathbf{x}Input or feature vector
∑\sumSum over all specified terms
LLSample count or loss function; check context

Reference-quality follow-up

Explore this calculation further
Deep Learning Book ↗ A reference textbook connecting machine-learning foundations, regularization, optimization, and convolution. CS231n Course Notes ↗ Connects classification, losses, validation, and CNNs through worked calculations.
Modern Artificial Intelligence I · MAI8-013 Identifier: MAI8-013

#8. The batch-size debate

The lecture introduces Keskar and colleagues’ ICLR 2017 findings: an empirical observation that large batches can approach sharper minimizers and generalize worse.

However, “sharp minima always generalize poorly” is not a theorem. A 2017 response already showed reparameterizations changing sharpness without changing the function, and 2025 research also reported sharp minima that generalize well.

A more careful 2026 reading is:

  • Evaluate batch size together with gradient noise, throughput, memory, and learning-rate schedules.
  • Sharpness is one useful diagnostic but can depend on parameterization.
  • Large-scale training jointly optimizes hardware utilization and optimization stability, rather than assuming smaller batches are always better.

#9. Connecting MIDAS research topics to the lecture

The lecture closes with examples of MIDAS Lab’s research directions.

  • Self-supervised and unsupervised learning
  • Iterative AI and iterative neural networks
  • Multimodal analysis
  • Nonconvex optimization
  • Compressed sensing
  • X-ray CT, MRI, PET, SPECT
  • Camera imaging, 3D reconstruction, and super-resolution
  • Vision-based autonomous systems
  • Digital healthcare

The first version presents improved RMSE in self-supervised regression for low-dose CT alongside neural view synthesis and NeRF-family examples. Read these as connections between foundational mathematics and actual research.

#10. 2026 update: large-scale training in practice

The lecture’s SGD equations remain a starting point, with many systems techniques layered above them in large-scale training.

#10.1 Optimizer

AdamW, which decouples weight decay from gradient updates, is widely used. The source’s PyTorch 2.13 discussion includes Muon alongside AdamW in the official optimizer API.

#10.2 Mixed precision

The source describes NVIDIA Transformer Engine 2.16 as supporting FP8 on Hopper, Ada, and Blackwell, with MXFP8 and NVFP4 also documented for Blackwell. Modern training therefore combines mathematical gradient descent, low-precision numerical representations, and scale management.

#10.3 Compilation and distributed training

PyTorch 2.13 provides torch.compile, FSDP/FSDP2, tensor parallelism, and distributed checkpoints in one ecosystem.

#11. Reconnecting Chapter 1 in one line

Represent data as vectors and matrices → Model it probabilistically → Construct functions with neural networks → Define losses → Differentiate to obtain gradients → Optimize repeatedly with minibatches → Validate generalization.

This progression is the core of Chapter 1.

Connect