The supremum of a set S⊂R is a value a∈Re such that ∀s∈S,s≤a and if b∈Re such that ∀s∈S,s≤b, then a≤b.
Supremum is essentially the “least upper bound” in a set. It always exists, and is called supS. The opposite of supremum is the infinimum.
Definition 3
The infinimum of a set S⊂R is a value a∈Re such that ∀s∈S,s≥a and if b∈Re such that ∀s∈S,s≥b, then a≥b.
The infinimum is the “greatest upper bound”. Like the supremum, it always exists, and it is denoted infS. Supremum and Infinimum can be applied to scalar function f:S→R by letting
supx∈Sf(x)=sup{f(x)∣x∈S}.
Norms
Definition 4
Let V be a vector space of R, then ∥⋅∥:V→R is a norm if ∀x,y∈V,α∈R,
∥x∥≥0,x=0⇔∥x∥=0,∥αx∥=∣α∣∥x∥,∥x+y∥≤∥x∥+∥y∥.
Definition 5
A normed space (V,∥⋅∥) is a vector space which is equipped with a norm ∥⋅∥:V→R.
Definition 6
The induced norm can be thought of as the maximum gain of the operator.
Definition 7
Sets
Definition 8
Definition 9
Open sets have a boundary which is not included in the set. By convention, we say that the empty set is open.
The opposite of an open set is a closed set.
Definition 10
Closed sets have a boundary which is included in the set.
Convergence
Definition 11
Convergence means that we can always find a finite time such that after that time, all points in the sequence stay within a specified norm ball.
Definition 12
A Cauchy sequence has a looser type of convergence than a convergent sequence since it only requires all elements to in the sequence to be part of the same norm ball after some time instead of requiring the sequence to get closer and closer to a single point.
Theorem 1
Definition 13
Because a complete space requires that Cauchy sequences converge, all cauchy sequences are convergent in a complete space. Two important complete spaces are
Every finite dimensional vector space
A complete normed space is also called a Banach Space.
Contractions
Definition 14
Definition 15
Informally, a contraction is a function which makes distances smaller. Suppose we look at a sequence defined by iterates of a function
Theorem 2 (Contraction Mapping Theorem)
The contraction mapping theorem proves that contractions have a unique fixed points, and that repeatedly applying the contraction will converge to the fixed point.
Continuity
Definition 16
\label{thm:continuity}
We can make the definition of continuity more restrictive by restraining the rate of growth of the function.
Definition 17
Theorem 3
This captures the idea of growing slower than linear in high dimensional space.
Definition 18
If we have an operator A which takes vectors from normed space (X,∥⋅∥X) and outputs vectors in normed space (Y,∥⋅∥Y), then we can define another norm on the vector space of operators from X→Y.
Let A:X→Y be an operator between normed spaces (X,∥⋅∥X) and (Y,∥⋅∥Y), then the induced norm of A is
∥A∥i=sup∥x∥X=0∥x∥X∥Ax∥Y
Two norms ∥⋅∥ and ∣∣∣⋅∣∣∣ on a vector space V are said to be equivalent if ∃k1,k2>0 such that
∀x∈V,k1∥x∥≤∣∣∣x∣∣∣≤k2∥x∥
If V is a finite dimensional vector space if and only if all norms of V are equivalent.
Let (V,∥⋅∥) be a normed space, a∈R, a>0, x0∈V, then the open ball of radius a centered around x0 is given by
Ba(x0)={x∈V∣∥x−x0∥<a}
A set S⊂V is open if ∀s0∈S,∃ϵ>0 such that Bϵ(s0)⊂S.
A set S is closed if ∼Sis open.
A sequence of points xk in normed space (V,∥⋅∥) converges to a point xˉ if
∀ϵ>0,∃N<∞, such that ∀k≥N,∥xk−xˉ∥<ϵ
A sequence xk is cauchy if
∀ϵ>0,∃N<∞ such that ∀n,m≥N,∥xm−xn∥<ϵ
If xn is a convergent sequence, then xnis a also a Cauchy sequence.
A normed space (V,∥⋅∥) is complete if every Cauchy sequence converges to a point in V.
(C[a,b],∥⋅∥∞), the set of continuously differentiable functions on the closed interval [a,b] equipped with the infinity norm.
A point x∗ is a fixed point of a function P:X→X if P(x∗)=x∗.
A function P:X→X is a contraction if ∃c∈R,0≤c<1 such that
∀x,y∈X,∥P(x)−P(y)∥≤c∥x−y∥
xk+1=P(xk)
where P is a function P:X→X. When does this sequence converge, and to what point will it converge?
If P:X→X is a contraction on the Banach space (X,∥⋅∥), then there is a unique x∗∈X such that P(x∗)=x∗ and ∀x0∈X, the sequence xn+1=P(xn) converges to x∗.
A function h:V→W on normed spaces (V,∥⋅∥V) and (W,∥⋅∥W) is continuous at a point x0 if ∀ϵ>0,∃δ>0 such that
∥x−x0∥V<δ⟹∥h(x)−h(x0)∥W<ϵ
Continuity essentially means that given an ϵ−ball in W, we can find a δ−ball in V which is mapped to the ball in W. If a function is continuous at all points x0, then we say the function is continuous.
A function h:V→W on normed spaces (V,∥⋅∥V) and (W,∥⋅∥W) is Lipschitz continuous at x0∈V if ∃r>0 and L<∞ such that
∀x,y∈Br(x0),∥h(x)−h(y)∥W≤L∥x−y∥V
A good interpretation of Lipschitz Continuity is that given two points in a ball around x0, the slope of the line connecting those two points is less than L. It means that the function is growing slower than linear for some region around x0. Lipschitz continuity implies continuity. If a function is lipschitz continuous with respect to one norm, it is also lipschitz continuous with respect to all equivalent norms.
When the function h is a function on Rn and is also differentiable, then Lipschitz continuity is easy to determine.
For a differentiable function h:Rn→Rn,
∃r>0,L<∞,x0∈Rn,∀x∈Br(x0),∂x∂h2≤L
implies Lipschitz Continuity at x0.
A function h:R→V is piecewise continuous if ∀k∈Z, h:[−k,k]→V is continuous except at a possibly finite number of points, and at the points of discontinuity ti, lims→0+h(ti+s) and lims→0−h(ti+s)exist and are finite.