Real Analysis

Definition 1

The extended real line is the set

$\{-\infty\} \cup \mathbb{R} \cup \{\infty\}.$

Definition 2

The supremum of a set $S \subset \mathbb{R}$ is a value $a \in \mathbb{R}_e$ such that $\forall s\in S,\ s \leq a$ and if $b \in \mathbb{R}_e$ such that $\forall s\in S,\ s \leq b$, then $a \leq b$.

Supremum is essentially the “least upper bound” in a set. It always exists, and is called $\sup S$. The opposite of supremum is the infinimum.

Definition 3

The infinimum of a set $S \subset \mathbb{R}$ is a value $a \in \mathbb{R}_e$ such that $\forall s\in S,\ s \geq a$ and if $b \in \mathbb{R}_e$ such that $\forall s\in S,\ s \geq b$, then $a \geq b$.

The infinimum is the “greatest upper bound”. Like the supremum, it always exists, and it is denoted $\inf S$. Supremum and Infinimum can be applied to scalar function $f: S\to \mathbb{R}$ by letting

$\sup_{x\in S} f(x) = \sup \{f(x) | x\in S \}.$

Norms

Definition 4

Let $V$ be a vector space of $\mathbb{R}$, then $\|\cdot\|: V \to \mathbb{R}$ is a norm if $\forall \boldsymbol{x},\boldsymbol{y}\in V, \alpha \in \mathbb{R}$,

$\|\boldsymbol{x}\| \geq 0, \qquad \boldsymbol{x} = 0 \Leftrightarrow \|\boldsymbol{x}\| = 0, \qquad \|\alpha \boldsymbol{x}\| = |\alpha|\|\boldsymbol{x}\|, \qquad \|\boldsymbol{x} + \boldsymbol{y}\| \leq \|\boldsymbol{x}\| + \|\boldsymbol{y}\|.$

Definition 5

A normed space $(V, \|\cdot\|)$ is a vector space which is equipped with a norm $\|\cdot\|: V \to \mathbb{R}$.

If we have an operator $A$ which takes vectors from normed space $(X, \|\cdot\|_X)$ and outputs vectors in normed space $(Y, \|\cdot\|_Y)$, then we can define another norm on the vector space of operators from $X\to Y$.

Definition 6

Let $A:X\to Y$ be an operator between normed spaces $(X, \|\cdot\|_X)$ and $(Y, \|\cdot\|_Y)$, then the induced norm of $A$ is

$\|A\|_i = \sup_{\|\boldsymbol{x}\|_X \neq 0} \frac{\|A\boldsymbol{x}\|_Y}{\|\boldsymbol{x}\|_X}$

The induced norm can be thought of as the maximum gain of the operator.

Definition 7

Two norms $\|\cdot\|$ and $|||\cdot|||$ on a vector space $V$ are said to be equivalent if $\exists k_1, k_2 > 0$ such that

$\forall \boldsymbol{x}\in V,\ k_1\|\boldsymbol{x}\| \leq |||\boldsymbol{x}||| \leq k_2\|\boldsymbol{x}\|$

If $V$ is a finite dimensional vector space if and only if all norms of $V$ are equivalent.

Sets

Definition 8

Let $(V, \|\cdot\|)$ be a normed space, $a\in \mathbb{R}$, $a > 0$, $\boldsymbol{x}_0\in V$, then the open ball of radius $a$ centered around $x_0$ is given by

$B_a(\boldsymbol{x}_0) = \{ \boldsymbol{x} \in V \ | \ \|\boldsymbol{x} - \boldsymbol{x}_0\| < a \}$

Definition 9

A set $S\subset V$ is open if $\forall \boldsymbol{s}_0\in S,\ \exists \epsilon > 0$ such that $B_\epsilon(\boldsymbol{s}_0) \subset S$.

Open sets have a boundary which is not included in the set. By convention, we say that the empty set is open.

The opposite of an open set is a closed set.

Definition 10

A set $S$ is closed if $\sim S$is open.

Closed sets have a boundary which is included in the set.

Convergence

Definition 11

A sequence of points $\boldsymbol{x}_k$ in normed space $(V, \|\cdot\|)$ converges to a point $\bar{\boldsymbol{x}}$ if

$\forall \epsilon > 0,\ \exists N < \infty,\ \text{ such that } \forall k \geq N, \|\boldsymbol{x}_k - \bar{\boldsymbol{x}}\| < \epsilon$

Convergence means that we can always find a finite time such that after that time, all points in the sequence stay within a specified norm ball.

Definition 12

A sequence $\boldsymbol{x}_k$ is cauchy if

$\forall \epsilon > 0,\ \exists N < \infty \text{ such that } \forall n,m \geq N, \|\boldsymbol{x}_m - \boldsymbol{x}_n\| < \epsilon$

A Cauchy sequence has a looser type of convergence than a convergent sequence since it only requires all elements to in the sequence to be part of the same norm ball after some time instead of requiring the sequence to get closer and closer to a single point.

Theorem 1

If $\boldsymbol{x}_n$ is a convergent sequence, then $\boldsymbol{x}_n$is a also a Cauchy sequence.

Definition 13

A normed space $(V, \|\cdot\|)$ is complete if every Cauchy sequence converges to a point in $V$.

Because a complete space requires that Cauchy sequences converge, all cauchy sequences are convergent in a complete space. Two important complete spaces are

1. Every finite dimensional vector space

2. $(C[a,b], \|\cdot\|_\infty)$, the set of continuously differentiable functions on the closed interval $[a,b]$ equipped with the infinity norm.

A complete normed space is also called a Banach Space.

Contractions

Definition 14

A point $\boldsymbol{x}^*$ is a fixed point of a function $P:X\to X$ if $P(\boldsymbol{x}^*)=\boldsymbol{x}^*$.

Definition 15

A function $P:X\to X$ is a contraction if $\exists c\in\mathbb{R}, 0 \leq c < 1$ such that

$\forall \boldsymbol{x},\boldsymbol{y}\in X,\ \|P(\boldsymbol{x}) - P(\boldsymbol{y})\| \leq c \|\boldsymbol{x}-\boldsymbol{y}\|$

Informally, a contraction is a function which makes distances smaller. Suppose we look at a sequence defined by iterates of a function

$\boldsymbol{x}_{k+1} = P(\boldsymbol{x}_k)$

where $P$ is a function $P:X\to X$. When does this sequence converge, and to what point will it converge?

Theorem 2 (Contraction Mapping Theorem)

If $P:X\to X$ is a contraction on the Banach space $(X, \|\cdot\|)$, then there is a unique $\boldsymbol{x}^*\in X$ such that $P(\boldsymbol{x}^*) = \boldsymbol{x}^*$ and $\forall \boldsymbol{x}_0\in X$, the sequence $\boldsymbol{x}_{n+1} = P(\boldsymbol{x}_n)$ converges to $\boldsymbol{x}^*$.

The contraction mapping theorem proves that contractions have a unique fixed points, and that repeatedly applying the contraction will converge to the fixed point.

Continuity

Definition 16

A function $h:V\to W$ on normed spaces $(V, \|\cdot\|_V)$ and $(W, \|\cdot\|_W)$ is continuous at a point $\boldsymbol{x}_0$ if $\forall \epsilon > 0, \exists \delta > 0$ such that

$\|\boldsymbol{x}-\boldsymbol{x}_0\|_V < \delta \implies \|h(\boldsymbol{x}) - h(\boldsymbol{x_0})\|_W < \epsilon$

\label{thm:continuity}

Continuity essentially means that given an $\epsilon-$ball in $W$, we can find a $\delta-$ball in $V$ which is mapped to the ball in $W$. If a function is continuous at all points $\boldsymbol{x}_0$, then we say the function is continuous.

We can make the definition of continuity more restrictive by restraining the rate of growth of the function.

Definition 17

A function $h:V\to W$ on normed spaces $(V, \|\cdot\|_V)$ and $(W, \|\cdot\|_W)$ is Lipschitz continuous at $\boldsymbol{x}_0\in V$ if $\exists r > 0$ and $L < \infty$ such that

$\forall \boldsymbol{x}, \boldsymbol{y}\in B_r(\boldsymbol{x}_0),\ \|h(\boldsymbol{x}) - h(\boldsymbol{y})\|_W \leq L \|\boldsymbol{x} - \boldsymbol{y}\|_V$

A good interpretation of Lipschitz Continuity is that given two points in a ball around $\boldsymbol{x}_0$, the slope of the line connecting those two points is less than $L$. It means that the function is growing slower than linear for some region around $\boldsymbol{x}_0$. Lipschitz continuity implies continuity. If a function is lipschitz continuous with respect to one norm, it is also lipschitz continuous with respect to all equivalent norms.

When the function $h$ is a function on $\mathbb{R}^n$ and is also differentiable, then Lipschitz continuity is easy to determine.

Theorem 3

For a differentiable function $h:\mathbb{R}^n\to\mathbb{R}^n$,

$\exists r>0, L < \infty, \boldsymbol{x}_0\in\mathbb{R}^n,\ \forall \boldsymbol{x}\in B_r(\boldsymbol{x}_0), \left\lvert\left\lvert\frac{\partial h}{\partial \boldsymbol{x}}\right\rvert\right\rvert_2 \leq L$

implies Lipschitz Continuity at $\boldsymbol{x}_0$.

This captures the idea of growing slower than linear in high dimensional space.

Definition 18

A function $h:\mathbb{R}\to V$ is piecewise continuous if $\forall k\in \mathbb{Z}$, $h:[-k, k] \to V$ is continuous except at a possibly finite number of points, and at the points of discontinuity $t_i$, $\lim_{s\to0^+} h(t_i+s)$ and $\lim_{s\to0^-}h(t_i+s)$exist and are finite.