Linear Algebra

Definition 1

Norms

Definition 2

Definition 3

Similar to vectors, matrices can also have norms.

Definition 4

Definition 5

Definition 6

The operator norms is defined as

Inner Products

Definition 7

In general, we can bound the absolute value of the standard inner product between two vectors.

Theorem 1 (Holder Inequality)

Functions

Definition 8

Definition 9

Definition 10

Definition 11

Definition 12

The half-spaces are the regions of space which a hyper-plane separates.

Definition 13

When a polyhedron is bounded, it is called a polytope.

Types of Functions

Theorem 2

Theorem 3

Any quadratic function can be written as the sum of a quadratic term involving a symmetric matrix and an affine term:

Another special class of functions are polyhedral functions.

Definition 14

Vector Calculus

We can also do calculus with vector functions.

Definition 15

Definition 16

The Hessian is akin to the second derivative in a 1D function. Note that the Hessian is a symmetric matrix.

Matrices

Theorem 4 (Fundamental Theorem of Linear Algebra)

Symmetric Matrices

Theorem 5 (Spectral Theorem)

Any symmetric matrix is orthogonally similar to a real diagonal matrix.

Definition 17

Theorem 6

Two special types of symmetric matrices are those with non-negative eigenvalues.

Definition 18

Definition 19

These matrices are important because they often have very clear geometric structures. For example, an ellipsoid in multi-dimensional space can be defined as the set of points

QR Factorization

Similar to how spectral theorem allows us to decompose symmetric matrices, QR factorization is another matrix decomposition technique that works for any general matrix.

Definition 20

Putting this in matrix form, we can see that

Singular Value Decomposition

Definition 21

A dyad is a rank-one matrix. It turns out that all matrices can be decomposed into a sum of dyads.

Definition 22

Writing the SVD tells us that

The Frobenius norm and spectral norm are tightly related to the SVD.

Last updated