Links

Real Analysis

Definition 1

The extended real line is the set
{}R{}.\{-\infty\} \cup \mathbb{R} \cup \{\infty\}.

Definition 2

The supremum of a set
SRS \subset \mathbb{R}
is a value
aRea \in \mathbb{R}_e
such that
sS, sa\forall s\in S,\ s \leq a
and if
bReb \in \mathbb{R}_e
such that
sS, sb\forall s\in S,\ s \leq b
, then
aba \leq b
.
Supremum is essentially the “least upper bound” in a set. It always exists, and is called
supS\sup S
. The opposite of supremum is the infinimum.

Definition 3

The infinimum of a set
SRS \subset \mathbb{R}
is a value
aRea \in \mathbb{R}_e
such that
sS, sa\forall s\in S,\ s \geq a
and if
bReb \in \mathbb{R}_e
such that
sS, sb\forall s\in S,\ s \geq b
, then
aba \geq b
.
The infinimum is the “greatest upper bound”. Like the supremum, it always exists, and it is denoted
infS\inf S
. Supremum and Infinimum can be applied to scalar function
f:SRf: S\to \mathbb{R}
by letting
supxSf(x)=sup{f(x)xS}.\sup_{x\in S} f(x) = \sup \{f(x) | x\in S \}.

Norms

Definition 4

Let
VV
be a vector space of
R\mathbb{R}
, then
:VR\|\cdot\|: V \to \mathbb{R}
is a norm if
x,yV,αR\forall \boldsymbol{x},\boldsymbol{y}\in V, \alpha \in \mathbb{R}
,
x0,x=0x=0,αx=αx,x+yx+y.\|\boldsymbol{x}\| \geq 0, \qquad \boldsymbol{x} = 0 \Leftrightarrow \|\boldsymbol{x}\| = 0, \qquad \|\alpha \boldsymbol{x}\| = |\alpha|\|\boldsymbol{x}\|, \qquad \|\boldsymbol{x} + \boldsymbol{y}\| \leq \|\boldsymbol{x}\| + \|\boldsymbol{y}\|.

Definition 5

A normed space
(V,)(V, \|\cdot\|)
is a vector space which is equipped with a norm
:VR\|\cdot\|: V \to \mathbb{R}
.
If we have an operator
AA
which takes vectors from normed space
(X,X)(X, \|\cdot\|_X)
and outputs vectors in normed space
(Y,Y)(Y, \|\cdot\|_Y)
, then we can define another norm on the vector space of operators from
XYX\to Y
.

Definition 6

Let
A:XYA:X\to Y
be an operator between normed spaces
(X,X)(X, \|\cdot\|_X)
and
(Y,Y)(Y, \|\cdot\|_Y)
, then the induced norm of
AA
is
Ai=supxX0AxYxX\|A\|_i = \sup_{\|\boldsymbol{x}\|_X \neq 0} \frac{\|A\boldsymbol{x}\|_Y}{\|\boldsymbol{x}\|_X}
The induced norm can be thought of as the maximum gain of the operator.

Definition 7

Two norms
\|\cdot\|
and
|||\cdot|||
on a vector space
VV
are said to be equivalent if
k1,k2>0\exists k_1, k_2 > 0
such that
xV, k1xxk2x\forall \boldsymbol{x}\in V,\ k_1\|\boldsymbol{x}\| \leq |||\boldsymbol{x}||| \leq k_2\|\boldsymbol{x}\|
If
VV
is a finite dimensional vector space if and only if all norms of
VV
are equivalent.

Sets

Definition 8

Let
(V,)(V, \|\cdot\|)
be a normed space,
aRa\in \mathbb{R}
,
a>0a > 0
,
x0V\boldsymbol{x}_0\in V
, then the open ball of radius
aa
centered around
x0x_0
is given by
Ba(x0)={xV  xx0<a}B_a(\boldsymbol{x}_0) = \{ \boldsymbol{x} \in V \ | \ \|\boldsymbol{x} - \boldsymbol{x}_0\| < a \}

Definition 9

A set
SVS\subset V
is open if
s0S, ϵ>0\forall \boldsymbol{s}_0\in S,\ \exists \epsilon > 0
such that
Bϵ(s0)SB_\epsilon(\boldsymbol{s}_0) \subset S
.
Open sets have a boundary which is not included in the set. By convention, we say that the empty set is open.
The opposite of an open set is a closed set.

Definition 10

A set
SS
is closed if
S\sim S
is open.
Closed sets have a boundary which is included in the set.

Convergence

Definition 11

A sequence of points
xk\boldsymbol{x}_k
in normed space
(V,)(V, \|\cdot\|)
converges to a point
xˉ\bar{\boldsymbol{x}}
if
ϵ>0, N<,  such that kN,xkxˉ<ϵ\forall \epsilon > 0,\ \exists N < \infty,\ \text{ such that } \forall k \geq N, \|\boldsymbol{x}_k - \bar{\boldsymbol{x}}\| < \epsilon
Convergence means that we can always find a finite time such that after that time, all points in the sequence stay within a specified norm ball.

Definition 12

A sequence
xk\boldsymbol{x}_k
is cauchy if
ϵ>0, N< such that n,mN,xmxn<ϵ\forall \epsilon > 0,\ \exists N < \infty \text{ such that } \forall n,m \geq N, \|\boldsymbol{x}_m - \boldsymbol{x}_n\| < \epsilon
A Cauchy sequence has a looser type of convergence than a convergent sequence since it only requires all elements to in the sequence to be part of the same norm ball after some time instead of requiring the sequence to get closer and closer to a single point.

Theorem 1

If
xn\boldsymbol{x}_n
is a convergent sequence, then
xn\boldsymbol{x}_n
is a also a Cauchy sequence.

Definition 13

A normed space
(V,)(V, \|\cdot\|)
is complete if every Cauchy sequence converges to a point in
VV
.
Because a complete space requires that Cauchy sequences converge, all cauchy sequences are convergent in a complete space. Two important complete spaces are
  1. 1.
    Every finite dimensional vector space
  2. 2.
    (C[a,b],)(C[a,b], \|\cdot\|_\infty)
    , the set of continuously differentiable functions on the closed interval
    [a,b][a,b]
    equipped with the infinity norm.
A complete normed space is also called a Banach Space.

Contractions

Definition 14

A point
x\boldsymbol{x}^*
is a fixed point of a function
P:XXP:X\to X
if
P(x)=xP(\boldsymbol{x}^*)=\boldsymbol{x}^*
.

Definition 15

A function
P:XXP:X\to X
is a contraction if
cR,0c<1\exists c\in\mathbb{R}, 0 \leq c < 1
such that
x,yX, P(x)P(y)cxy\forall \boldsymbol{x},\boldsymbol{y}\in X,\ \|P(\boldsymbol{x}) - P(\boldsymbol{y})\| \leq c \|\boldsymbol{x}-\boldsymbol{y}\|
Informally, a contraction is a function which makes distances smaller. Suppose we look at a sequence defined by iterates of a function
xk+1=P(xk)\boldsymbol{x}_{k+1} = P(\boldsymbol{x}_k)
where
PP
is a function
P:XXP:X\to X
. When does this sequence converge, and to what point will it converge?

Theorem 2 (Contraction Mapping Theorem)

If
P:XXP:X\to X
is a contraction on the Banach space
(X,)(X, \|\cdot\|)
, then there is a unique
xX\boldsymbol{x}^*\in X
such that
P(x)=xP(\boldsymbol{x}^*) = \boldsymbol{x}^*
and
x0X\forall \boldsymbol{x}_0\in X
, the sequence
xn+1=P(xn)\boldsymbol{x}_{n+1} = P(\boldsymbol{x}_n)
converges to
x\boldsymbol{x}^*
.
The contraction mapping theorem proves that contractions have a unique fixed points, and that repeatedly applying the contraction will converge to the fixed point.

Continuity

Definition 16

A function
h:VWh:V\to W
on normed spaces
(V,V)(V, \|\cdot\|_V)
and
(W,W)(W, \|\cdot\|_W)
is continuous at a point
x0\boldsymbol{x}_0
if
ϵ>0,δ>0\forall \epsilon > 0, \exists \delta > 0
such that
xx0V<δ    h(x)h(x0)W<ϵ\|\boldsymbol{x}-\boldsymbol{x}_0\|_V < \delta \implies \|h(\boldsymbol{x}) - h(\boldsymbol{x_0})\|_W < \epsilon
\label{thm:continuity}
Continuity essentially means that given an
ϵ\epsilon-
ball in
WW
, we can find a
δ\delta-
ball in
VV
which is mapped to the ball in
WW
. If a function is continuous at all points
x0\boldsymbol{x}_0
, then we say the function is continuous.
We can make the definition of continuity more restrictive by restraining the rate of growth of the function.

Definition 17

A function
h:VWh:V\to W
on normed spaces
(V,V)(V, \|\cdot\|_V)
and
(W,W)(W, \|\cdot\|_W)
is Lipschitz continuous at
x0V\boldsymbol{x}_0\in V
if
r>0\exists r > 0
and
L<L < \infty
such that
x,yBr(x0), h(x)h(y)WLxyV\forall \boldsymbol{x}, \boldsymbol{y}\in B_r(\boldsymbol{x}_0),\ \|h(\boldsymbol{x}) - h(\boldsymbol{y})\|_W \leq L \|\boldsymbol{x} - \boldsymbol{y}\|_V
A good interpretation of Lipschitz Continuity is that given two points in a ball around
x0\boldsymbol{x}_0
, the slope of the line connecting those two points is less than
LL
. It means that the function is growing slower than linear for some region around
x0\boldsymbol{x}_0
. Lipschitz continuity implies continuity. If a function is lipschitz continuous with respect to one norm, it is also lipschitz continuous with respect to all equivalent norms.
When the function
hh
is a function on
Rn\mathbb{R}^n
and is also differentiable, then Lipschitz continuity is easy to determine.

Theorem 3

For a differentiable function
h:RnRnh:\mathbb{R}^n\to\mathbb{R}^n
,
r>0,L<,x0Rn, xBr(x0),hx2L\exists r>0, L < \infty, \boldsymbol{x}_0\in\mathbb{R}^n,\ \forall \boldsymbol{x}\in B_r(\boldsymbol{x}_0), \left\lvert\left\lvert\frac{\partial h}{\partial \boldsymbol{x}}\right\rvert\right\rvert_2 \leq L
implies Lipschitz Continuity at
x0\boldsymbol{x}_0
.
This captures the idea of growing slower than linear in high dimensional space.

Definition 18

A function
h:RVh:\mathbb{R}\to V
is piecewise continuous if
kZ\forall k\in \mathbb{Z}
,
h:[k,k]Vh:[-k, k] \to V
is continuous except at a possibly finite number of points, and at the points of discontinuity
tit_i
,
lims0+h(ti+s)\lim_{s\to0^+} h(t_i+s)
and
lims0h(ti+s)\lim_{s\to0^-}h(t_i+s)
exist and are finite.