> For the complete documentation index, see [llms.txt](https://aparande.gitbook.io/berkeley-notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://aparande.gitbook.io/berkeley-notes/eecs225a-0/eecs225a-1.md).

# Hilbert Space Theory

Complex random variables form a Hilbert space with inner product $$\langle X, Y \rangle = \mathbb{E}\left\[XY^\*\right]$$. If we have a random complex vector, then we can use Hilbert Theory in a more efficient manner by looking at the matrix of inner products. For simplicity, we will call this the “inner product” of two complex vectors.

{% hint style="info" %}

#### Definition 1

Let the inner product between two random, complex vectors $$\boldsymbol{Z\_1}, \boldsymbol{Z\_2}$$ be defined as

$$\langle \boldsymbol{Z\_1}, \boldsymbol{Z\_2} \rangle = \mathbb{E}\left\[\boldsymbol{Z\_1}\boldsymbol{Z\_2}^\*\right]$$
{% endhint %}

The ij-th entry of the matrix is simply the scalar inner product $$\mathbb{E}\left\[X\_iY\_j^\*\right]$$where $$X\_i$$ and $$Y\_j$$ are the ith and jth entries of $$\boldsymbol{X}$$ and $$\boldsymbol{Y}$$ respectively. This means the matrix is equivalent to the cross correlation $$R\_{XY}$$ between the two vectors. We can also specify the auto-correlation $$R\_X = \langle \boldsymbol{X}, \boldsymbol{X} \rangle$$and auto-covariance $$\Sigma\_X = \langle \boldsymbol{X} - \mathbb{E}\left\[\boldsymbol{X}\right] , \boldsymbol{X} - \mathbb{E}\left\[\boldsymbol{X}\right] \rangle$$. One reason why we can think of this matrix as the inner product is because it also satisfies the properties of inner products. In particular, it is

1. Linear: $$\langle \alpha\_1\boldsymbol{V\_1}+\alpha\_2\boldsymbol{V\_2}, \boldsymbol{u} \rangle = \alpha\_1\langle \boldsymbol{V\_1}, \boldsymbol{u} \rangle + \alpha\_2\langle \boldsymbol{V\_2}, \boldsymbol{u} \rangle$$.
2. Reflexive: $$\langle \boldsymbol{U}, \boldsymbol{V} \rangle = \langle \boldsymbol{V}, \boldsymbol{U} \rangle ^\*$$.
3. Non-degeneracy: $$\langle \boldsymbol{V}, \boldsymbol{V} \rangle = \boldsymbol{0} \Leftrightarrow \boldsymbol{V} = \boldsymbol{0}$$.

Since we are thinking of the matrix as an inner product, we can also think of the norm as a matrix.

{% hint style="info" %}

#### Definition 2

The norm of a complex random vector is given by $$|\boldsymbol{Z}|^2 = \langle \boldsymbol{Z}, \boldsymbol{Z} \rangle$$.
{% endhint %}

When thinking of inner products as matrices instead of scalars, we must rewrite the Hilbert Projection Theorem to use matrices instead.

{% hint style="info" %}

#### Theorem 1 (Hilbert Projection Theorem) <a href="#theorem-1" id="theorem-1"></a>

The minimization problem $$\min\_{\hat{\boldsymbol{X}}(\boldsymbol{Y})}|\hat{\boldsymbol{X}}(\boldsymbol{Y}) - \boldsymbol{X}|^2$$ has a unique solution which is a linear function of $$\boldsymbol{Y}$$. The error is orthogonal to the linear subspace of $$\boldsymbol{Y}$$ (i.e $$\langle \boldsymbol{X} - \hat{\boldsymbol{X}}, \boldsymbol{Y} \rangle = \boldsymbol{0}$$)
{% endhint %}

When we do a minimization over a matrix, we are minimizing it in a PSD sense, so for any other linear function $$\boldsymbol{X}'$$,

$$|\boldsymbol{X}-\hat{\boldsymbol{X}}|^2 \preceq |\boldsymbol{X} - \boldsymbol{X}'|^2.$$

## Innovations

Suppose we have jointly distributed random variables $$Y\_0, Y\_1,\cdots,Y\_n$$. Ideally, we would be able to “de-correlate” them so each new vector $$E\_0$$ captures the new information which is orthogonal to previous random vectors in the sequence. Since vectors of a Hilbert Space operate like vectors in $$\mathbb{R}^n$$, we can simply do Gram-Schmidt on the $${Y\_i}\_{i=0}^n$$.

{% hint style="info" %}

#### Definition 3

Given jointly distributed random vectors $${Y\_i}\_{i=0}^n$$ with $$\mathcal{L}*i = \text{span}{Y\_j}*{j=0}^i$$, the ith innovation $$E\_i$$ is given by

$$E\_i = Y\_i - \text{proj}(Y\_i|\mathcal{L}*{i-1}) = Y\_i - \sum*{j=0}^{i-1}\frac{\langle Y\_i, E\_j \rangle }{|E\_j|^2}E\_j$$
{% endhint %}

Innovations have two key properties.

1. $$\forall i\neq j,\ \langle E\_i, E\_j \rangle =0$$
2. $$\forall i,\ \text{span}{Y\_j}*{j=0}^i = \text{span}{E\_j}*{j=0}^i$$

We can also write innovations in terms of a matrix where $$\boldsymbol{\varepsilon} = A\boldsymbol{Y}$$ where $$\boldsymbol{\varepsilon} = \begin{bmatrix}E\_0 & E\_1 & \cdots & E\_n\end{bmatrix}^T$$ and $$\boldsymbol{Y} = \begin{bmatrix}Y\_0 & Y\_1 & \cdots & Y\_n\end{bmatrix}^T$$. Since each $$E\_i$$ only depends on the previous $$Y\_i$$, then A must be lower triangular, and because we need each $$E\_i$$ to be mutually orthogonal, $$R\_{\varepsilon}$$ should be diagonal. $$R\_{\varepsilon} = AR\_YA^*$$, so if $$R\_Y \succ 0$$, then we can use its unique LDL decomposition $$R\_Y = LDL^*$$ and let $$A = L^{-1}$$.